Method and device for image restoration

Abstract

A method for image restoration, according to one aspect, may comprise the steps of: acquiring a single dataset including a plurality of raw images including noise; estimating noise statistics by performing spatial correlation analysis among the raw images included in the single dataset; determining a blind-spot size on the basis of the noise statistics; training a blind-spot neural network having a structure corresponding to the determined blind-spot size, in a self-supervised manner by using the single dataset; and generating a restored image from the raw images by using the blind-spot neural network.

Description

Image restoration methods and devices

[0001] The present disclosure relates to an image restoration method and an image restoration device.

[0002]

[0003] In modern precision imaging fields, including medical diagnostic imaging (X-ray CT, MRI, etc.), space and astronomical observation, bio live-cell fluorescence microscopy, and scanning electron microscopes (SEM) and transmission electron microscopes (TEM) for semiconductor nanoprocess inspection, technology that analyzes the fine structural characteristics and dynamic changes of the observed object at high resolution plays an important role.

[0004] Despite the continuous advancement of imaging hardware, limitations exist in that the waves or particles themselves, such as photons or electron beams used for measurement, can cause effects such as radiation exposure, phototoxicity, or structural damage to the observed material or biological tissue. To minimize such damage, low-illumination or low-dose imaging techniques that limit the incident signal are required.

[0005] However, in low-dose environments, Poisson noise inevitably occurs as the absolute number of signal particles reaching the detector becomes insufficient. Furthermore, the physical limitations of pixelated detectors, coincidence loss, and mechanical or thermal drift that may occur during the time-series image acquisition process act as factors causing structural noise and blur in the acquired raw data.

[0006] Various traditional image denoising techniques, such as Gaussian filtering, bandpass filtering, Wiener filtering, and BM3D, have been proposed to remove such mechanical and physical noise. However, these techniques have limitations in that they can reduce fine structural details and valid information of the observed object during the noise removal process.

[0007] To address this, AI neural network-based deep learning denoising technologies are being introduced; however, these technologies require noise-free ground truth images from which noise has been removed during the training process. Yet, in environments such as actual human body internal imaging, dynamic biological observation settings, or nanoscale material observation, securing such ground truth images is not easy due to dose limitations and the temporal responsiveness of the subjects.

[0008] Furthermore, the statistical characteristics of noise and the spatial diffusion patterns between adjacent pixels can vary depending on imaging conditions, lighting environments, and lens and detector settings. Consequently, when a denoising model pre-trained under specific conditions is applied to a different observation environment, there is a possibility that reconstruction performance may degrade or distortion may occur.

[0009] Therefore, there is a need for a new image restoration technique that can preserve the structural information of the original while estimating noise characteristics from a single acquired raw data point without relying on a ground truth reference image.

[0010]

[0011] The object of the present disclosure is to provide an image restoration method and an image restoration device capable of solving the problem of physical limitations in acquiring a ground-truth reference image under low-light or low-dose imaging conditions applied to prevent damage to an object in observation environments utilizing wave or particle beams, such as medical imaging, observation of living cells, astronomical observation, and transmission electron microscopy (TEM) analysis.

[0012] The object of the present disclosure is to provide an image restoration method and an image restoration device capable of separating noise patterns by utilizing a single raw image data containing noise itself as a training input and an inference target, without a separate clean training dataset.

[0013] Another objective of the present disclosure is to provide an image restoration method and an image restoration device capable of estimating the statistical characteristics of noise that change according to the type of imaging equipment, the measured dose ratio, the detector characteristics, and the drift state of the observed sample in a data-dependent manner through spatial correlation analysis between data tensors.

[0014] Another objective of the present disclosure is to provide an image restoration method and an image restoration device capable of suppressing overfitting in which noise patterns are identically learned in a neural network and reducing the loss of structural information.

[0015] Another objective of the present disclosure is to provide an image restoration method and an image restoration device capable of improving the accuracy and spatiotemporal resolution of structural restoration in time-series-based imaging environments, such as dynamic cell observation, medical contrast agent tracking, and nanoparticle kinetics analysis.

[0016] In light of one or more defects, disadvantages, and / or problems mentioned in the background art, the present disclosure is proposed to achieve one or more objectives, including solving or mitigating these problems.

[0017] The purpose of this disclosure is not limited to any specific purpose or problem to be solved. Other purposes and technical problems to be solved by this disclosure will be clearly understood by those skilled in the art from the following detailed description, the accompanying drawings, and the combination of the various embodiments disclosed in the claims.

[0018] These purposes and technical advantages of the present disclosure may be realized through one or more components described in the appended claims and logical combinations thereof. Furthermore, it is clarified that not all components described in the claims must be necessarily included in all embodiments of the present disclosure.

[0019]

[0020] An image restoration method according to one embodiment may include the steps of: acquiring a single dataset containing a plurality of raw images containing noise; performing spatial correlation analysis between raw images included in the single dataset to estimate noise statistics; determining a blind spot size based on the noise statistics; training a blind spot neural network configured according to the determined blind spot size in a self-supervised manner using the single dataset; and generating a restored image from the raw images using the blind spot neural network.

[0021] In one embodiment, the step of estimating noise statistics by performing spatial correlation analysis between raw images included in the single dataset may include: moving a raw image tensor selected from the single dataset in pixel units in a first direction and a second direction, respectively; calculating a correlation value between the moved tensor and the original tensor; generating a two-dimensional spatial correlation heatmap using the correlation value; measuring a spatial diffusion radius in which a correlation value greater than or equal to a preset threshold is maintained in the two-dimensional spatial correlation heatmap; and determining a physical blocking area in the form of N×N (where N is an odd number greater than or equal to 1) pixels as the blind spot size in proportion to the number of pixels of the measured spatial diffusion radius.

[0022] In one embodiment, the blindspot neural network may include a dilution convolutional layer. The expansion rate of the dilution convolutional layer may be set based on the determined blindspot size.

[0023] In one embodiment, raw images included in the single dataset may be provided as training inputs for the blindspot neural network and as inference targets for calculating the loss function.

[0024] In one embodiment, the step of training the blindspot neural network in a self-supervised manner may include the step of extracting a patch having a preset pixel size from a raw image included in the single dataset, the step of augmenting the training data by applying a geometric transformation to the patch, and the step of self-updating parameters by backpropagating the loss in a direction that minimizes the mean squared error (MSE) between the prediction result tensor of the blindspot neural network and the raw image assigned as the inference target.

[0025] An image restoration device according to one embodiment may include a data storage that stores one or more instructions and a processor that executes said instructions.

[0026] In one embodiment, the processor may be configured to acquire a single dataset containing a plurality of raw images containing noise, perform spatial correlation analysis between the raw images included in the single dataset to estimate noise statistics, determine a blind spot size based on the noise statistics, train a blind spot neural network configured according to the determined blind spot size in a self-supervised manner using the single dataset, and generate a reconstructed image from the raw images using the blind spot neural network.

[0027] In one embodiment, the processor may be configured to perform the spatial correlation analysis by moving a raw image tensor selected from the single dataset in pixel units in a first direction and a second direction, respectively, calculating a correlation value between the moved tensor and the original tensor, generating a two-dimensional spatial correlation heatmap using the correlation value, measuring a spatial diffusion radius in which a correlation value greater than or equal to a preset threshold is maintained in the two-dimensional spatial correlation heatmap, and determining a physical blocking area in the form of N×N (where N is an odd number greater than or equal to 1) pixels as the blind spot size in proportion to the number of pixels of the measured spatial diffusion radius.

[0028] In one embodiment, the blindspot neural network may include a dilution convolutional layer. The expansion rate of the dilution convolutional layer may be set based on the determined blindspot size.

[0029] In one embodiment, raw images included in the single dataset may be provided as training inputs for the blindspot neural network and as inference targets for calculating the loss function.

[0030] In one embodiment, the processor may be configured to self-update parameters by backpropagating the loss in a direction that minimizes the mean squared error between the prediction result tensor of the blindspot neural network and the raw image assigned to the inference target when training the blindspot neural network.

[0031] The aforementioned components, steps, and their functional relationships are merely exemplary means illustrating one or more embodiments described in the claims of this disclosure. Without departing from the technical spirit and scope of this disclosure, the aforementioned components or steps may be replaced, omitted, or combined in various ways. Furthermore, the technical scope claimed by this disclosure is not limited to any combination of all components and steps described in the claims.

[0032]

[0033] First, the image restoration method and device according to the embodiments completely eliminate the biggest barrier to entry in conventional technology, which requires separately securing a defect-free ground-truth dataset in advance to train an artificial intelligence model, consuming vast amounts of time and resources. Instead, they provide an innovative self-supervised learning framework that self-updates parameters using only a single raw image dataset that inherently contains instrumental and physical noise. Through this, they possess excellent technical scalability, enabling universal high-throughput, damage-free noise restoration without the cost of collecting separate reference data in a wide range of industrial and medical imaging fields. These fields include low-dose medical X-ray CT and in situ transmission electron microscopes (TEM), which have the inherent limitation of being unable to acquire ground-truth images, as well as high-sensitivity space and astronomical observation images, low-light biofluorescence microscopes, and scanning electron microscopes (SEMs) for inspecting ultrafine defects on semiconductor wafers.

[0034] Second, the image restoration method and device according to the embodiments completely depart from the conventional uniform method of blindly applying fixed filters or general-purpose neural network models pre-trained in specific environments. Instead, whenever a new raw image dataset is acquired, spatial correlation analysis is performed within the single data to precisely estimate the physical dominance range of the noise, and a customized blind spot size (e.g., 3×3, 5×5, 9×9, etc.) that perfectly corresponds to this is dynamically and variably formed. By physically blocking the inflow of target pixel information within the reception area based on the customized blind spot derived in this way, the phenomenon of the neural network being identically overfitted to a specific noise pattern can be completely prevented. As a result, compared to existing single-image or video denoising self-supervised models (Noise2Void, UDVD, etc.), an improvement in peak signal-to-noise ratio (PSNR) of up to approximately 20 decibels (dB) and a dramatic numerical improvement in structural similarity (SSIM) reaching 0.9 are achieved. In addition, the image restoration method and device according to the embodiments provide a significant image quality improvement effect that perfectly removes only noise without loss of effective structural information of human tissue, cells, celestial bodies, or microscopic atomic units.

[0035]

[0036] FIG. 1 is a block diagram showing the configuration of an image restoration device according to an exemplary embodiment.

[0037] FIG. 2 is a flowchart showing the overall flow of an image restoration method according to an exemplary embodiment.

[0038] FIG. 3 is a flowchart showing the detailed flow of the step of estimating noise statistics in an image restoration method according to an exemplary embodiment.

[0039] Figure 4 shows an example of a two-dimensional spatial correlation heatmap appearing in an image restoration method according to an exemplary embodiment.

[0040] FIG. 5 is a flowchart showing the detailed flow of the step of determining the blind spot size in an image restoration method according to an exemplary embodiment.

[0041] FIG. 6 is a flowchart showing the detailed flow of the step of training a blindspot neural network in a self-supervised manner in an image restoration method according to an exemplary embodiment.

[0042] Figure 7 is a diagram showing the internal operation structure of a blindspot neural network according to an exemplary embodiment and the structural and resulting differences with a general convolution operation module.

[0043] FIG. 8 shows an actual experimental result image comparing the performance of an image restoration method according to an exemplary embodiment with that of the prior art and comparative examples.

[0044] FIG. 9 is a graph comparing the quantitative performance indicators of an image restoration method according to an exemplary embodiment with the prior art.

[0045]

[0046] The features, aspects, and advantages of the present disclosure will become more apparent with reference to the following detailed description and the accompanying drawings. To enable a person skilled in the art to easily understand and practice the technical concept of the present disclosure, the following detailed description includes exemplary embodiments and methods of operation for explaining the present disclosure.

[0047] The following description and drawings are provided merely as examples and descriptions to illustrate the spirit and scope of the present disclosure, and are not intended to limit the present disclosure. Accordingly, the scope of the present disclosure should not be unduly limited by these exemplary embodiments.

[0048] The components, steps, operations, actions, and arrangements and combinations of data values ​​used in this disclosure are intended to provide specific examples for exemplary embodiments. In certain embodiments, some components, steps, operations, actions, and / or arrangements and combinations of data values ​​may be omitted, modified, combined, or replaced. Additionally, detailed descriptions of known technologies, known manufacturing methods, and configurations related to this disclosure may be omitted if it is determined that doing so would unnecessarily obscure or obscure the essence of this disclosure.

[0049] In the following description and the accompanying drawings, the same reference numerals indicate identical or functionally similar components or parts. In the description of specific embodiments, multiple drawings may be referenced simultaneously.

[0050] FIG. 1 is a block diagram showing the configuration of an image restoration device (1) according to an exemplary embodiment.

[0051] Referring to FIG. 1, the image restoration device (1) may include a processor (10) and a data storage (20).

[0052] The image restoration device (1) can acquire a single dataset from an external device. The single dataset may include a plurality of raw images. At least one of the plurality of raw images may include noise. In an exemplary embodiment, the raw image may be an image acquired without a separate ground-truth image. That is, the single dataset may not include a reference image from which noise has been removed.

[0053] At least one of the multiple raw images may contain various types of noise generated during the shooting process. The noise may include at least one of Poisson noise, signal dispersion across pixel boundaries, and noise caused by sample drift. Such noise may form spatial correlations between adjacent pixels. The spatial correlation of the noise may vary depending on the type of image and shooting conditions.

[0054] For example, the raw image may include an image acquired by at least one of Transmission Electron Microscopy (TEM), Scanning Transmission Electron Microscopy (STEM), and cryo-electron tomography (cryo-ET). For example, the raw image may include an image acquired by liquid cell in situ Transmission Electron Microscopy (liquid cell in situ TEM). For example, the raw image may include an image acquired by time-series Scanning Transmission Electron Microscopy (time-series STEM). For example, the raw image may include an image acquired by at least one of an optical microscope, a fluorescence microscope, a low-dose X-ray CT, an astronomical observation camera, and Scanning Electron Microscopy (SEM). That is, the raw image included in a single dataset input to the image restoration device (1) according to the exemplary embodiment is not limited to an image acquired by a specific imaging device. An image restoration device (1) according to an exemplary embodiment can be applied to any domain in which a single dataset including a plurality of raw images containing noise can be input.

[0055] The raw image may include a time-series image. The time-series image may include multiple image frames acquired consecutively in chronological order for the same subject. Each frame constituting the time-series image may include a noise pattern that has a spatial correlation with other temporally adjacent frames.

[0056] The raw image may originate from a video captured by a shooting device connected to an image restoration device (1). The shooting device may be connected to the image restoration device (1) via a wired or wireless connection. The video may include multiple frames captured continuously in chronological order. The raw image may be an image frame extracted from at least some of the multiple frames constituting the video. That is, a single dataset may include multiple image frames extracted from the video.

[0057] The number of frames extracted from the video is not particularly limited. For example, all frames constituting the video can be used as a single dataset. Some of the frames constituting the video can be subsampled at predetermined frame intervals and used as a single dataset. In this way, since a single dataset can be formed using only the frames subsampled from the video, the image restoration device (1) according to the present disclosure can be applied to datasets of limited size.

[0058] Image frames originating from a video may contain noise patterns that have spatial correlations between temporally adjacent frames. Sample drift occurring during the video recording process can cause asymmetry in the noise patterns between adjacent frames. Since the image restoration device (1) according to the present disclosure can estimate various noise statistics including such asymmetric noise patterns through spatial correlation analysis, it can be effectively applied to image frames originating from a video.

[0059] The imaging device may include at least one of a transmission electron microscope (TEM), a scanning transmission electron microscope (STEM), a cryo-electron tomography (cryo-ET), an optical microscope, a fluorescence microscope, an X-ray CT device, and a scanning electron microscope (SEM). The types of imaging devices are not limited thereto and may include any device capable of capturing video containing noise.

[0060] In another embodiment, the image restoration device (1) may obtain a single dataset from a device other than the shooting device.

[0061] The number of raw images included in a single dataset is not particularly limited. A single dataset may consist of only a small number of frames. For example, a single dataset may include 20 or more frames. A single dataset may include 100 frames. As such, the image restoration device (1) according to the present disclosure does not require a large dataset, so the cost and time of data collection can be reduced.

[0062] A single dataset containing a plurality of raw images and / or a plurality of raw images can be stored in data storage (20).

[0063] The processor (10) can execute one or more instructions stored in the data storage (20). The processor (10) may be configured to acquire a single dataset, perform spatial correlation analysis to estimate noise statistics, determine a blind spot size based on the noise statistics, train a blind spot neural network configured according to the determined blind spot size in a self-supervised manner using the single dataset, and generate a reconstructed image from a raw image using the trained blind spot neural network. The processor (10) may dynamically train a new blind spot neural network dependent on the dataset whenever a new single dataset is input. Details regarding each operation of the processor (10) will be described later with reference to FIGS. 2 through 6.

[0064] The processor (10) may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a combination thereof. Since the training and inference operations of a blindspot neural network are computationally intensive tasks, it may be desirable for the processor (10) to include a GPU. However, the specific type of processor (10) is not limited thereto and may include any computational unit capable of executing instructions.

[0065] The processor (10) can support automatic mixed precision computation using tensor cores. The processor (10) can dramatically reduce the computation time required for training and inference of a single dataset. The image restoration device (1) can continuously restore large electron microscope images with high-throughput in real-time or near-real-time. Since the image restoration device (1) does not need to maintain or search a vast external training database, it can maximize the space efficiency of the data storage (20).

[0066] Data storage (20) can store one or more instructions executed by the processor (10). Data storage (20) can store a raw image dataset. Data storage (20) can store model parameters calculated during the training process of the blindspot neural network. Data storage (20) can store a two-dimensional spatial correlation heatmap generated by spatial correlation analysis. Data storage (20) can store determined blindspot size information.

[0067] The data storage (20) may include at least one of a hard disk drive (HDD), a solid-state drive (SSD), flash memory, RAM (Random Access Memory), and combinations thereof. The specific type of data storage (20) is not limited thereto and may include any storage medium capable of storing data and providing it to the processor (10).

[0068] The data storage (20) can store model parameters calculated during the training process of the blindspot neural network. The model parameters may include the weights and biases of each layer constituting the blindspot neural network. The model parameters can be repeatedly updated during the training process in a direction that minimizes the loss function. After training is completed, the model parameters at the point in time that indicates the optimal validation loss can be stored in the data storage (20). The stored model parameters can be loaded by the processor (10) during the restoration image generation step and used for inference of the blindspot neural network.

[0069] A neural network is a computational structure composed of multiple nodes and connections between nodes, capable of approximating a function that generates output data from input data. A neural network may include an input layer, one or more hidden layers, and an output layer. Each layer may include multiple nodes. Each node can generate an output value by multiplying an input value by a weight, adding a bias, and then applying an activation function. A neural network can learn a desired input-output relationship through a training process that iteratively updates weights and biases in a direction that minimizes the loss function based on training data.

[0070] A Convolutional Neural Network (CNN) is a type of neural network that may include convolutional layers that perform convolution operations. Convolutional layers can extract spatial features by applying kernels to input feature maps. Since CNNs can effectively process the spatial structure of image data, they can be widely utilized in various image processing fields, including image restoration, image classification, and object detection.

[0071] A blindspot neural network is a type of convolutional neural network that may have a blindspot structure that blocks information about the pixel to be inferred and its adjacent pixels. A blindspot neural network can predict the noise-removed value of a pixel by utilizing only information from surrounding pixels without directly referencing the pixel to be inferred. Such a blindspot structure can enable self-supervised training using the same raw image as both the training input and the inference target.

[0072] The model parameters of the blindspot neural network can be individually optimized for each dataset during the training process. That is, the blindspot neural network can generate a denoising model having unique model parameters that reflect the noise characteristics of each dataset. The model parameters of the blindspot neural network after training can be stored in a data storage (20), and the processor (10) can load the stored model parameters to perform inference to generate a restored image from a raw image.

[0073] Although not illustrated, the data storage (20) may include a model parameter storage area for storing model parameters of a blindspot neural network. The model parameter storage area may be a logically or physically separated storage space within the data storage (20). Model parameters stored in the model parameter storage area may be read by the processor (10) and used for inference of the blindspot neural network. The specific implementation method of the model parameter storage area is not particularly limited and may be implemented in various ways that are obvious to a person skilled in the art.

[0074] The processor (10) and the data storage (20) may be interconnected via a bus, an interface, or a communication channel. The processor (10) may read instructions and data from the data storage (20). The processor (10) may write the results of operations to the data storage (20).

[0075] The image restoration device (1) can output a restored image based on a raw image included in a single dataset. The restored image may be an image from which noise has been removed from the raw image using a trained blindspot neural network. The restored image may have an improved Signal-to-Noise Ratio (SNR) compared to the raw image. The restored image may have an improved Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) compared to the raw image. The restored image may be stored in a data storage (20). The restored image may also be provided to another device (not shown).

[0076] Although not illustrated, the image restoration device (1) may further include a communication interface for receiving a raw image from an external shooting device. Although not illustrated, the image restoration device (1) may further include an output interface for transmitting a restored image to an external device. The specific types of the communication interface and the output interface are not particularly limited and may include various wired or wireless communication means that are obvious to a person skilled in the art.

[0077] FIG. 2 is a flowchart showing the overall flow of an image restoration method according to an exemplary embodiment.

[0078] Referring to FIG. 2, an image restoration method according to an exemplary embodiment may include the steps of: acquiring a single dataset containing a plurality of raw images containing noise (S100); performing spatial correlation analysis between raw images included in the single dataset to estimate noise statistics (S200); determining a blind spot size based on the noise statistics (S300); training a blind spot neural network configured according to the determined blind spot size in a self-supervised manner using the single dataset (S400); and generating a restored image from the raw images using the blind spot neural network (S500).

[0079] In step S100, the processor (10) of the image restoration device (1) may acquire a single dataset containing a plurality of raw images containing noise. The single dataset may not include a ground-truth image from which noise has been removed. This may mean that the image restoration method according to the present disclosure can be performed using only a self-guided method without collecting ground-truth images. Collecting ground-truth images separately is often practically extremely difficult or impossible due to signal insufficient signal in low-dose shooting environments, sample drift, and the diversity of shooting conditions. The image restoration method according to the present disclosure can overcome such limitations. The single dataset may be provided to the processor (10) from an external device or data storage (20). The specific details of the single dataset are as described above with reference to FIG. 1.

[0080] In step S200, the processor (10) can estimate noise statistics by performing spatial correlation analysis between raw images included in a single dataset. The noise statistics may include information regarding the range of spatial influence of noise on adjacent pixels within the raw images. The noise statistics may vary depending on the image capture conditions and the type of capturing device.

[0081] In this specification, 'noise statistics' may comprehensively refer to the mathematical and statistical characteristics of noise inherently embedded within a raw image. Noise statistics may refer to an indicator that quantifies the spatial propagation effect of noise generated by the instrumental limitations or physical factors of the imaging process on surrounding adjacent pixels. Noise statistics may include information regarding spatial dependency and variance correlation between noise pixels. Depending on the type of imaging equipment, the set dose ratio, the physical characteristics of the detector, or the drift state of the sample, noise statistics may appear randomly different for each imaging session. The processor (10) can precisely estimate the noise statistics unique to a single dataset in a data-dependent manner through spatial correlation analysis.

[0082] In this specification, "estimation of noise statistics" may refer to a series of computational processes that quantify unknown noise distribution characteristics by mathematically inversely calculating them using only raw data in an uncertain environment where a ground truth image is completely absent. The processor (10) can precisely compute spatial correlations between pixels within the raw image tensor. The processor (10) can generate a two-dimensional spatial correlation heatmap based on the computed correlations. The processor (10) can analyze the shape and diffusion radius of the generated two-dimensional spatial correlation heatmap. Through heatmap analysis, the processor (10) can indirectly determine the range and direction of the physical ripple effect of noise on adjacent pixels. Thus, the entire algorithmic process of mathematically determining the dominance radius of invisible noise through spatial correlation analysis of a single dataset itself without separate external reference data can be defined as the operation of estimating noise statistics.

[0083] Conventional self-guided denoising methods assume that noise statistics are known in advance or use only a single-pixel-sized blind spot. However, in real-world environments, such as electron microscope environments, noise statistics are often unknown, and a single-pixel-sized blind spot cannot sufficiently eliminate inter-pixel noise correlations. The image restoration method according to the present disclosure can be used even when noise statistics are not known in advance by performing spatial correlation analysis individually for each dataset to estimate noise statistics. The details of step S200 will be described later with reference to FIG. 3.

[0084] In step S300, the processor (10) can determine the blind spot size based on the noise statistics estimated in step S200. The blind spot size may be a value that defines the spatial range of pixels that the blind spot neural network cannot reference during inference in the form of N×N (where N is an odd number greater than or equal to 1) pixels. The blind spot size may be automatically determined based on the spatial diffusion radius of the noise. If the blind spot size is determined to be smaller than the spatial diffusion radius of the noise, the blind spot neural network may not be able to sufficiently block the noise pattern, and noise overfitting may occur. By determining the blind spot size to be suitable for the spatial diffusion radius of the noise, effective denoising can be performed without noise overfitting.

[0085] In the present disclosure, the spatial diffusion radius of noise may refer to the physical limit distance over which mechanical or physical noise, initially generated at a single pixel location, spreads to adjacent surrounding pixel areas. The spatial diffusion radius of noise may comprehensively refer to a pixel-unit range in which a noise signal spreads with spatial dependency due to electron beam scattering, the resolution limit of a detector sensor, the spreading of a point spread function, or sample drift phenomena occurring during imaging. Adjacent pixels located within the spatial diffusion radius of noise may statistically have strong correlations and data dependencies with each other. The process of accurately measuring the spatial diffusion radius of noise may be essential to fundamentally prevent the overfitting phenomenon in which a blindspot neural network mistakes the noise patterns of surrounding pixels for valid information and learns the identity function.

[0086] In the present disclosure, the spatial diffusion radius of the noise may refer to the maximum pixel range in which correlation values ​​greater than or equal to a preset threshold are continuously maintained within a two-dimensional spatial correlation heatmap. The preset threshold may refer to a mathematical lower limit value of the correlation value at which the noise is determined to have a statistically significant interference effect on adjacent pixels. The preset threshold may be flexibly set differently in a data-dependent manner depending on the type of imaging equipment, lens conditions, and examples of the observation environment. The processor (10) may finally determine the spatial diffusion radius of the noise by calculating the pixel distance from the center of the two-dimensional spatial correlation heatmap to the boundary point where the significant correlation value is interrupted, based on the preset threshold.

[0087] The details of step S300 will be described later with reference to Fig. 4.

[0088] In step S400, the processor (10) can train a blindspot neural network configured according to the blindspot size determined in step S300 in a self-supervised manner using a single dataset. Self-supervised training can be performed by simultaneously providing raw images included in the single dataset as training inputs for the blindspot neural network and as inference targets for calculating the loss function. That is, the blindspot neural network can be trained without separate ground truth images. Details of step S400 will be described later with reference to FIG. 5. The structure of the blindspot neural network will be described later with reference to FIG. 6.

[0089] In step S500, the processor (10) can generate a reconstructed image from the raw image using the blindspot neural network trained in step S400. The reconstructed image may be an image from which noise contained in the raw image has been removed. The reconstructed image may have at least one of an enhanced SNR, PSNR, and SSIM compared to the raw image. The generated reconstructed image may be stored in the data storage (20). The generated reconstructed image may be output to the outside of the image restoration device (1).

[0090] Steps S100 to S500 illustrated in FIG. 2 can be performed in sequence. Steps S100 to S500 can be executed by a processor (10). Instructions corresponding to steps S100 to S500 can be stored in a data storage (20).

[0091] FIG. 3 is a flowchart showing the detailed flow of the step of estimating noise statistics in an image restoration method according to an exemplary embodiment.

[0092] In an exemplary embodiment, step S200 may include a step of selecting and standardizing a raw image tensor (S210), a step of shifting the raw image tensor in pixel units (S220), a step of calculating a correlation value between the shifted tensor and the original tensor (S230), and a step of generating a two-dimensional spatial correlation heatmap (S240).

[0093] In step S210, the processor (10) may select a raw image from a single dataset. The processor (10) may convert the selected raw image into an image tensor. The processor (10) may standardize the image tensor. Standardization may include a process of adjusting the mean of the image tensor to 0 and the variance to 1. Through standardization, the range of pixel values ​​of the image tensor can be normalized to increase the accuracy of the subsequent correlation value calculation. Step S210 is a preprocessing step for spatial correlation analysis and may have a direct effect on the accuracy of steps S220 through S230.

[0094] In step S220, the processor (10) can move the standardized raw image tensor in pixel units in a first direction and a second direction, respectively. For example, the first direction may be the horizontal direction (x direction) of the image, and the second direction may be the vertical direction (y direction) of the image. The processor (10) can move the image tensor sequentially by one pixel in the first direction. The processor (10) can move the image tensor sequentially by one pixel in the second direction. The range of movement may be set differently depending on the expected spatial range of influence of the noise.

[0095] In step S230, the processor (10) can calculate the correlation between the tensor moved in step S220 and the original tensor. The correlation can be calculated for two image tensors x and y by the following formula.

[0096]

[0097]

[0098]

[0099] Here, x i and y i represents the i-th pixel value of the original tensor and the shifted tensor, respectively, and <·> represents the average value. The correlation value can range from -1 to 1. A correlation value closer to 1 indicates a high degree of similarity between the noise patterns of the two tensors. A correlation value closer to 0 indicates independence between the noise patterns of the two tensors.

[0100] In an exemplary embodiment, the processor (10) can repeat steps S220 and S230 multiple times to calculate multiple correlation values ​​corresponding to each movement direction and movement distance.

[0101] In step S240, the processor (10) can aggregate multiple correlation values ​​calculated in step S230 to generate a two-dimensional spatial correlation heatmap. The two-dimensional spatial correlation heatmap visualizes the spatial range of influence that noise has on adjacent pixels in the form of a two-dimensional matrix. The two-dimensional spatial correlation heatmap may show a distribution in which the correlation value is high at the center and decreases as it moves away from the center. The two-dimensional spatial correlation heatmap may be stored in a data storage (20).

[0102] Noise statistics may be spatially correlated uniformly across the entire area within the image. A processor (10) can verify this by dividing the image into multiple sub-regions and individually calculating the noise correlation of each sub-region and comparing it with the noise correlation of the entire image. If the noise is spatially correlated uniformly across the entire image, it may be appropriate to use a 2D spatial correlation heatmap for the entire image as a criterion for determining the blind spot size.

[0103] Figure 4 shows an example of a two-dimensional spatial correlation heatmap appearing in an image restoration method according to an exemplary embodiment.

[0104] A two-dimensional spatial correlation heatmap can appear as symmetric or asymmetric depending on the shooting conditions. In FIG. 4, (A) shows an example of a symmetric heatmap, and (B) shows an example of an asymmetric heatmap. The structure and shape of the symmetric heatmap and the asymmetric heatmap may vary depending on the embodiment.

[0105] A symmetric heatmap can appear when the spatial influence of noise is uniformly distributed in the first and second directions. For example, in the case of raw images acquired in a silicon nitride liquid cell TEM environment, a symmetric heatmap can be generated in which noise exhibits high correlation within a distance of 4 pixels.

[0106] Asymmetric heatmaps can appear when the spatial influence of noise is skewed in a specific direction. For example, in a time-series STEM environment, noise patterns may appear asymmetrically due to sample drift occurring during the raster scan process, and a corresponding asymmetric heatmap can be generated.

[0107] The processor (10) does not need to perform complex and cumbersome image alignment or a separate drift correction algorithm beforehand. The direction and physical trajectory of the sample drift can be mathematically fully reflected in the 2D spatial correlation heatmap itself. The image restoration method according to the present disclosure can immediately absorb distorted noise patterns in a data-dependent manner and reflect them in the model.

[0108] The image restoration method according to the present disclosure can determine an appropriate blind spot size for both symmetric and asymmetric heatmaps, so it can be universally applied to raw images acquired under various shooting conditions.

[0109] When step S240 is completed, the processor (10) can perform step S310 of FIG. 5.

[0110] FIG. 5 is a flowchart showing the detailed flow of the step of determining the blind spot size in an image restoration method according to an exemplary embodiment.

[0111] Referring to FIG. 5, step S300 may include a step of measuring the spatial diffusion radius (S310) and a step of determining the blind spot size (S320). Step S300 may be performed by receiving a two-dimensional spatial correlation heatmap generated in step S240 of FIG. 3 as input.

[0112] In step S310, the processor (10) can analyze the two-dimensional spatial correlation heatmap generated in step S240 to measure the spatial diffusion radius of the noise. The spatial diffusion radius may refer to a range of pixels in the two-dimensional spatial correlation heatmap where a correlation value greater than or equal to a preset threshold is maintained. The preset threshold may refer to a lower limit of the correlation value at which the noise is determined to have a significant effect on adjacent pixels. The preset threshold may be set differently depending on the embodiment.

[0113] In an exemplary embodiment, the processor (10) can measure the maximum pixel distance from the center of the two-dimensional spatial correlation heatmap, in which a correlation value greater than or equal to a preset threshold is maintained in each of the first direction and the second direction, as a spatial diffusion radius.

[0114] In step S320, the processor (10) can determine the blind spot size in proportion to the number of pixels of the spatial diffusion radius measured in step S310.

[0115] In an exemplary embodiment, the processor (10) may determine the blind spot size in the form of N×N pixels (where N is an odd number greater than or equal to 1). In other words, the blind spot size may be determined as a minimum odd size that can encompass the spatial diffusion radius of the noise. For example, if the spatial diffusion radius is 0 pixels, the blind spot size may be determined as 1×1. If the spatial diffusion radius is approximately 1 pixel, the blind spot size may be determined as 3×3. If the spatial diffusion radius is approximately 2 pixels, the blind spot size may be determined as 5×5. If the spatial diffusion radius is approximately 4 pixels, the blind spot size may be determined as 9×9.

[0116] If the blind spot size is determined to be smaller than the spatial diffusion radius of the noise, the blind spot neural network may not be able to sufficiently block the noise, leading to noise overfitting. Conversely, if the blind spot size is determined to be excessively larger than the spatial diffusion radius of the noise, the blind spot neural network may unnecessarily block even valid structural information. In this case, the resolution of the restored image may be degraded, or a blur phenomenon may occur in which fine details are obscured. When the blind spot size is determined to be appropriate for the spatial diffusion radius of the noise, the blind spot neural network can effectively remove noise without noise overfitting. Thus, since the image restoration method according to the present disclosure automatically determines the blind spot size based on noise statistics, it is possible to determine an optimized blind spot size even under various shooting conditions where the noise statistics are not known in advance.

[0117] The spatial diffusion radius of noise may vary depending on the image acquisition conditions and the type of imaging device. For example, when using a charge-coupled device (CCD) or a direct electron detector (DED) in a linear acquisition mode, the received electronic signal may be dispersed more widely than the size of the physical sensor pixels, forming noise dependency between adjacent pixels. As another example, in STEM imaging, raster scanning and sample drift may alter the inter-pixel dependency of noise. The image restoration method according to the present disclosure can automatically determine the blind spot size by performing steps S200 and S300 individually for each dataset, and thus can be universally applied to various imaging conditions and imaging devices.

[0118] The determined blind spot size can be stored in data storage (20).

[0119] When step S320 is completed, the processor (10) can perform step S400 of FIG. 2.

[0120] FIG. 6 is a flowchart showing the detailed flow of the step of training a blindspot neural network in a self-supervised manner in an image restoration method according to an exemplary embodiment.

[0121] In an exemplary embodiment, raw images included in a single dataset may be provided as training inputs for a blindspot neural network while simultaneously serving as inference targets for calculating the loss function of the blindspot neural network. That is, the training inputs and inference targets may be provided from the same raw images rather than separate ground images. This self-supervised method enables the training of a blindspot neural network without the need to collect ground images separately.

[0122] In an exemplary embodiment, a blindspot neural network can avoid learning an identity function that simply copies the input, even when using the same raw image as both the training input and the inference target, through a blindspot structure that blocks information about the pixel to be inferred and its adjacent pixels. The blindspot neural network can be trained to predict the noise-removed value of a pixel by using only information from surrounding pixels without directly referencing the pixel to be inferred.

[0123] Referring to FIG. 6, step S400 may include a step of extracting a patch (S410), a step of applying a geometric transformation (S420), a step of forward propagating a blind spot neural network (S430), a step of calculating an MSE loss (S440), and a step of performing backpropagation (S450).

[0124] In step S410, the processor (10) can extract patches having a preset pixel size from raw images included in a single dataset.

[0125] A patch may be an area randomly cropped from a raw image. The pixel size of the patch is not specifically limited. For example, the pixel size of the patch may be 256×256 pixels. Extracting a patch through random cropping can suppress overfitting by preventing the blindspot neural network from completely remembering a specific noise pattern. The processor (10) can extract a new patch for each iteration of the training process.

[0126] In step S420, the processor (10) can augment training data for training a blindspot neural network by applying a geometric transformation to the patch extracted in step S410.

[0127] In an exemplary embodiment, geometric transformation may include at least one of rotation and inversion. Rotation may include rotating the patch by 90°. Inversion may include at least one of inverting the patch in a horizontal direction and inverting it in a vertical direction.

[0128] Data augmentation can improve the generalization performance of blindspot neural networks by generating more training data from a single dataset of limited size.

[0129] In other embodiments, step S420 may be omitted.

[0130] In step S430, the processor (10) can input training data into the blindspot neural network to perform forward propagation. The blindspot neural network can be configured according to the blindspot size (42) determined in step S300. The structure of the blindspot neural network will be described later with reference to FIG. 6.

[0131] A blindspot neural network can use one or more frames temporally adjacent to the target frame as input. For example, a blindspot neural network can use a total of five frames as input, including the target frame and four temporally adjacent frames. By utilizing the temporal information of adjacent frames, the prediction accuracy of the blindspot neural network can be improved. Through forward propagation, the blindspot neural network can generate a prediction result tensor containing the predicted value of the target pixel with noise removed.

[0132] In step S450, the processor (10) can calculate the mean squared error (MSE) loss between the prediction result tensor generated in step S440 and the raw image assigned as the inference target. The MSE loss can be calculated by the following formula.

[0133]

[0134]

[0135]

[0136] Here, represents the pixel value of the inference target corresponding to the i-th pixel of the j-th frame, and represents the predicted value of the blindspot neural network. The processor (10) can optimize the parameters of the blindspot neural network in a way that minimizes the MSE loss.

[0137] In step S460, the processor (10) can backpropagate the MSE loss calculated in step S450 to self-update the parameters of the blindspot neural network.

[0138] Parameter updates can be performed by an optimization algorithm. For example, the optimization algorithm may include a Rectified Adam (RAdam) optimizer. For example, the learning rate may be adjusted by a warm-up and cosine annealing scheduler. The types of optimization algorithms and specific types of learning rate schedulers are not limited thereto, and various optimization algorithms and learning rate schedulers that are obvious to a person of ordinary skill may be applied.

[0139] The minimization of the empirical risk of training can be defined by the following formula.

[0140]

[0141]

[0142]

[0143] Here, θ represents a parameter of the blindspot neural network (50), and represents the acceptance area input (blindspot constraints applied) for the i-th pixel of the j-th frame, and represents the inference target pixel value. The processor (10) can repeat step S460 to progressively optimize the parameter θ.

[0144] The processor (10) can determine whether the blindspot neural network (50) converges after step S460. Convergence can be determined based on the validation loss. For example, the processor (10) can calculate the validation loss by separating some images from a single dataset into a validation dataset. If the validation loss does not satisfy a preset criterion, the processor (10) can return to step S410 and repeat the training. If the validation loss satisfies the preset criterion, the processor (10) can terminate the training and store the model parameters at the time showing the optimal validation loss in the data storage (20).

[0145] The number of training epochs can be adjusted according to the size of the dataset. For example, training can be performed with 2,000 epochs for most datasets. For the Cryo-ET dataset, training can be performed with 100 epochs.

[0146] When step S460 is completed, a blindspot neural network can be generated. Accordingly, the processor (10) can perform step S500 of FIG. 2. That is, the processor (10) can generate a reconstructed image corresponding to the raw image using the blindspot neural network.

[0147] Figure 7 is a diagram showing the internal operation structure of a blindspot neural network according to an exemplary embodiment and the structural and resulting differences with a general convolution operation module.

[0148] In Fig. 7, the input grid (left), the dotted rectangle representing the convolution operation structure (center), and the predicted output result for the inference target (right) are shown divided into three rows.

[0149] The three dotted rectangles arranged in the center of FIG. 7 each refer to independent and individually driven operation modules (61a, 61b, 61c). Inside each operation module, a 3×3 convolution kernel (51) and a dilated convolution layer (53) may be optionally placed.

[0150] The first row of FIG. 7 shows the structure of a computation module (61a) to which no blindspot structure is applied. The computation module (61a) is a general convolutional computation module to which no blindspot constraints for information blocking are applied. A 3×3 convolutional kernel (51) is placed inside the computation module (61a). The computation module (61a) performs prediction by referencing the information of the entire receptive field of the network containing the inference target pixel without filtering. In this case, the computation module (61a) directly references the value of the inference target pixel itself. If the same raw image is used simultaneously as the training input and the inference target, the neural network containing the computation module (61a) learns an identity function that simply copies the input noise pixel value as is and outputs it. The output image (Identity) shown on the right side of the first row represents a failed restoration result in which the noise is not removed at all due to the learning of the identity function. As such, without a blindspot structure, it is impossible to learn a valid denoising model through self-supervised training.

[0151] The second row of FIG. 7 shows the structure of a computation module (61b) with a blind spot size (42a) set to 1×1 pixel. The computation module (61b) is a computation module with an ideal blind spot of 1×1 size applied. Inside the computation module (61b), a 3×3 convolutional kernel (51) and a dilution convolutional layer (53) are arranged. The computation module (61b) physically blocks only one pixel to be inferred. In the input grid on the left of row 2, it can be seen that only the 1×1 area corresponding to the pixel to be inferred is blocked. When the blind spot size (42a) is 1×1, the computation module (61b) blocks the value of the pixel to be inferred itself but references all values ​​of immediately adjacent pixels. If the noise of adjacent pixels is spatially strongly correlated with the noise of the pixel to be inferred and the spatial diffusion radius of the noise exceeds 1 pixel, the noise information is indirectly leaked into the computation module (61b) through the adjacent pixels that are allowed to be referenced. When noise information is leaked, the neural network including the computation module (61b) experiences a noise overfitting phenomenon in which it learns by mistaking the noise pattern itself for valid structural information. The output image (Low SNR) shown on the right side of row 2 shows an incomplete reconstruction result exhibiting a low signal-to-noise ratio (SNR) as noise overfitting occurs because the blind spot size (42a) is set smaller than the spatial diffusion radius of the actual noise.

[0152] The third row of FIG. 7 shows the structure of a computation module (61c) with a blind spot size (42b) set to 3×3 pixels. The computation module (61c) is a computation module with a 3×3 size Real blind spot applied. Inside the computation module (61c), a 3×3 convolutional kernel (51) and a dilution convolutional layer (53) are arranged. The computation module (61c) physically blocks information in a total 3×3 area, including the inference target pixel and 8 adjacent pixels surrounding the inference target pixel. In the input grid on the left of the third row, it can be seen that the entire 3×3 area is grouped into one large area and blocked. When the blind spot size (42b) is 3×3, the computation module (61c) structurally blocks information in all adjacent pixels existing within the spatial diffusion radius of the noise. As the inflow of information is blocked, the neural network containing the computation module (61c) fundamentally avoids noise overfitting and predicts the target pixel value using only valid structural information from the outer edge. The output image (High SNR) shown on the right side of row 3 represents the result of generating a high SNR restored image with noise removed by setting the blind spot size (42b) to fit the spatial spread radius of the noise.

[0153] Referring to the second and third rows of FIG. 7, the computation module (61b, 61c) includes a dilution convolutional layer (53). In an exemplary embodiment, the processor (10) can physically maintain the blind spot constraint while expanding the receiving area by controlling the dilation rate of the dilution convolutional layer (53).

[0154] The expansion rate of the diluted convolutional layer (53) can be set in conjunction with the blind spot size determined in step S300. As the expansion rate increases, the size of the blind spot may increase. For example, if the blind spot size is 1×1, the expansion rate may be set to 1. If the blind spot size is 3×3, the expansion rate may be set to 2. If the blind spot size is 5×5, the expansion rate may be set to 3. The specific value of the expansion rate is not limited to this and may be adjusted within a range obvious to a person skilled in the art according to the blind spot size.

[0155] In FIG. 7, the receptive field of the network represents the spatial range of the entire pixel area that each computation module (61a, 61b, 61c) references when inferring a single pixel. In FIG. 7, the electronic signal represents the scattered distribution of the raw signal containing the blurring effect, received at each pixel of the electron microscope or camera detector. The dilution convolution layer (53) collects context information valid for prediction from the wide receptive field outside the blocked blind spot area.

[0156] A blindspot neural network including computation modules (61b, 61c) uses only the remaining pixel information after blocking pixel information corresponding to the blindspot size from the receptive region xRF(i) including pixel i and its adjacent pixels when performing a prediction for a single pixel i. The function of the blindspot neural network including computation modules (61b, 61c) can be defined as follows.

[0157]

[0158]

[0159]

[0160] Here, θ represents the parameters of the blindspot neural network, and represents the predicted value of pixel i. The blindspot neural network predicts the noise-removed value of pixel i using only information from surrounding pixels, without directly referencing the information of pixel i. Due to this physical blocking structure, the blindspot neural network does not learn the identity function even when the same raw image is used simultaneously as the training input and the inference target.

[0161] As illustrated in FIG. 7, a reconstructed image with a high SNR is generated only when the blind spot size is determined to be suitable for the spatial diffusion radius of the noise (61c, 42b). This visually supports the technical significance of the image reconstruction method according to the present disclosure, which includes the step (S300) of automatically determining the blind spot size based on noise statistics.

[0162] Training of the blindspot neural network can be performed individually for each dataset without relying on external general-purpose data. The blindspot neural network can generate denoising model parameters optimized for the unique noise characteristics and diffusion radius of each dataset. Although not illustrated, the blindspot neural network may support mixed-precision operations to improve training and inference speeds. The specific implementation of the blindspot neural network is not limited to this and may be modified to the extent obvious to a person skilled in the art.

[0163] FIG. 8 shows an actual experimental result image comparing the performance of an image restoration method according to an exemplary embodiment with that of the prior art and comparative examples.

[0164] FIG. 8 shows the denoising results for actual raw data obtained in a graphene liquid cell environment. FIG. 8 shows, in order from left to right, a raw image containing noise (A), the result of applying a conventional general magnetic map model (N2V) (B), the result of a comparative example with the blind spot size set to 1x1 (C), and the result of optimizing the blind spot size to 3x3 based on noise statistics (D).

[0165] The dotted box indicated in the center of each image in Fig. 8 represents the region of interest where the nanoparticle being observed is actually located. The area inside the dotted box can be used as a visual indicator to evaluate how completely the fine atomic-level lattice structure has been restored without damage. Referring to the area inside the dotted box in the raw image (A) on the left side of Fig. 8, it can be seen that the shape and internal lattice structure of the nanoparticle cannot be identified at all due to extreme physical noise.

[0166] Referring to the result image (B) of the prior art, it can be seen that noise is not properly removed throughout the image and stain-like artifacts remain. If one looks inside the dotted box in the result image (B) of the prior art, it can be seen that the atomic lattice structure of the nanoparticles is distorted beyond recognition due to incomplete noise removal. The prior art fails to restore the structural information of the image because it does not consider the spatial correlation between pixels of the actual noise at all.

[0167] Referring to the result image (C) of the comparative example, it can be seen that severe noise overfitting occurred within the network when only a 1x1 blind spot was applied without considering the noise diffusion radius. If one looks inside the dotted box in the result image (C) of the comparative example, the result shows that noise particles remain intact to the extent that there is no significant visual difference from the initial raw image. This proves that denoising at the fine structure level is structurally impossible when blind spots smaller than the noise diffusion radius are applied collectively.

[0168] Referring to the result image (D) with the blind spot size optimized to 3x3, it can be intuitively seen that the overall noise has been completely removed by applying the customized blind spot of size 3x3 derived in the noise statistics estimation step. Referring to the area inside the dotted box in the result image (D) with the blind spot size optimized to 3x3, it can be clearly seen that while the surrounding background noise has been cleanly eliminated, the fine lattice structure at the atomic level constituting the nanoparticles has been restored very clearly without any damage.

[0169] The atomic lattice spacing restored by the embodiments of the present disclosure can perfectly match the theoretical actual physical lattice spacing of the material. The blindspot neural network can fundamentally suppress hallucinations that artificially generate virtual, fake structures not present in the raw image. The embodiments of the present disclosure can provide highly reliable restored images that can be directly utilized as a precise quantitative measure for scientific analysis.

[0170] Ultimately, according to an embodiment of the present disclosure, only valid structural information can be fully preserved and output in high quality.

[0171] As can be seen from the quantitative figures indicated in FIG. 8, according to the embodiment of the present disclosure, a signal-to-noise ratio that has dramatically increased compared to the comparative example is observed. According to the embodiment of the present disclosure, the step of dynamically and variably determining the blind spot size in accordance with the actual spatial diffusion radius of the noise leads to an improvement in image quality without destroying the microstructure inside the dotted box. According to the embodiment of the present disclosure, the best visual contrast and signal-to-noise ratio can be achieved by completely suppressing noise in the background portion of the raw image.

[0172] FIG. 9 is a graph comparing the quantitative performance indicators of an image restoration method according to an exemplary embodiment with the prior art.

[0173] The performance of an image restoration method can be quantitatively evaluated using PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) metrics.

[0174] PSNR is an indicator that represents the signal quality between a reconstructed image and a ground truth image in decibels (dB). A higher PSNR value indicates superior quality of the reconstructed image.

[0175] SSIM is an indicator that represents the structural similarity between the reconstructed image and the ground truth image as a value between 0 and 1. The closer the SSIM value is to 1, the higher the structural similarity between the reconstructed image and the ground truth image.

[0176] Referring to FIG. 9, the image restoration method according to the present disclosure (blind spot neural network, blind spot size 9×9 applied) can exhibit significantly improved PSNR and SSIM values ​​compared to a noise image, N2V, BM3D, UDVD, and UDVD*. For example, in a silicon nitride liquid cell TEM simulation dataset, the image restoration method according to the present invention can achieve a PSNR improvement of about 20 dB and an SSIM improvement of about 0.6 compared to a noise image. In a Pt nanoparticle simulation dataset on a ceria support, the image restoration method according to the present invention can achieve a PSNR improvement of about 20 dB and an SSIM improvement of about 0.9.

[0177] When the blind spot size is set smaller than the spatial diffusion radius of the noise, that is, when the blind spot size is set to 1×1 or 3×3, noise overfitting occurs, which may degrade PSNR and SSIM values. When the blind spot size is set to be suitable for the spatial diffusion radius of the noise, for example, when the blind spot size is set to 9×9, significantly improved PSNR and SSIM values ​​can be achieved. Such quantitative performance improvement can be achieved by the image restoration method according to the present disclosure, which automatically determines the blind spot size based on noise statistics and configures a blind spot neural network according to the determined blind spot size.

[0178] Such a dramatic leap in quantitative performance indicators corresponds to a significant effect that a person skilled in the art could never have easily predicted from conventional simple self-guided denoising techniques. The image restoration method according to the present disclosure can provide a technological advancement that fundamentally overcomes the limitations of existing techniques in extreme low-dose imaging environments where ground truth images are non-existent.

[0179] Various features and functions of the system, device, and method disclosed in the foregoing detailed description have been described in detail below through exemplary embodiments with reference to the attached drawings. In the drawings, similar symbols generally represent similar components unless otherwise indicated by the context.

[0180] With respect to some or all of the ladder diagrams, scenarios, and flowcharts in the drawings, each block and / or communication may represent information processing and / or information transmission according to exemplary embodiments. The scope of these exemplary embodiments includes alternative embodiments. In these alternative embodiments, functions described, for example, as blocks, transmissions, communications, requests, responses, and / or messages may be executed differently from the order shown or discussed according to the relevant functions, and may be executed substantially simultaneously or in reverse order. Additionally, more or fewer blocks and / or functions may be used with any of the ladder diagrams, scenarios, and flowcharts discussed in this disclosure, and these ladder diagrams, scenarios, and flowcharts may be combined with one another in part or in whole.

[0181] A block representing information processing may correspond to a circuit portion that can be configured to perform a specific logical function of the method or technique described in this disclosure. Alternatively or additionally, a block representing information processing may correspond to a module, segment, or part of program code (including related data). Program code may include one or more instructions executable by a processor to implement a specific logical function or operation in the method or technique. Program code and / or related data may be stored on any type of computer-readable medium, such as a storage device including a disk, a hard drive, or other storage media.

[0182] Computer-readable media may also include non-transient computer-readable media, which include non-transient computer-readable media that store data for a short period of time, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may also include non-transient computer-readable media that store data for a long period of time, which may be secondary or permanent long-term storage, such as read-only memory (ROM), optical or magnetic disks, and compact disc read-only memory (CD-ROM). Computer-readable media may also be other volatile or non-volatile storage systems. Computer-readable media may be considered, for example, as computer-readable storage media or tangible storage devices.

[0183] Additionally, a block representing one or more information transmissions may correspond to information transmission between software and / or hardware modules on the same physical device. However, other information transmissions may occur between software modules and / or hardware modules on different physical devices.

[0184] Although preferred or exemplary embodiments of the present disclosure have been described above, the scope of protection of the present disclosure should not be interpreted as being limited by these specific embodiments. It is obvious to those skilled in the art that various modifications and variations can be made to the contents of the present disclosure without departing from the basic concept and technical spirit of the present disclosure.

[0185] Such modifications and variations are intended to be included within the scope of this disclosure and should be defined by the appended claims and their equivalents. The arrangements, structures, devices, processes, methods, and materials of specific components described in this disclosure and illustrated in the appended drawings are merely exemplary embodiments and should not be construed as intended to limit the scope of this disclosure.

[0186] In particular, all structures, methods, and / or devices functionally equivalent to the components listed in the claims of this disclosure shall be deemed to fall within the scope of this disclosure, and such equivalents will be apparent to those skilled in the art from the above detailed description.

[0187] Accordingly, the scope of the present disclosure shall be determined solely by the appended claims, and terms, examples, and embodiments used in the detailed description shall not be interpreted in a manner that limits the scope of the claims. It is clarified that the present disclosure includes all combinations and sub-combinations of components that have not been previously disclosed or known.

Claims

1. A step of acquiring a single dataset containing a plurality of raw images containing noise; A step of estimating noise statistics by performing spatial correlation analysis between raw images included in the single dataset above; A step of determining the blind spot size based on the above noise statistics; A step of training a blind spot neural network configured according to the above-determined blind spot size in a self-supervised manner using the single dataset; and A method comprising the step of generating a reconstructed image from the raw image using the above blindspot neural network. Image restoration method.

2. In Paragraph 1, The step of estimating noise statistics by performing spatial correlation analysis between raw images included in the single dataset above A step of moving a raw image tensor selected from the single dataset in pixel units in a first direction and a second direction, respectively; and A step of calculating a correlation value between the shifted tensor and the original tensor; and A method comprising the step of generating a two-dimensional spatial correlation heatmap using the above correlation values. Image restoration method.

3. In Paragraph 2, The step of determining the blind spot size based on the above noise statistics is A step of measuring a spatial diffusion radius in which a correlation value greater than or equal to a preset threshold is maintained in the above two-dimensional spatial correlation heatmap; and A step comprising determining a physical blocking area in the form of N×N pixels (where N is an odd number greater than or equal to 1) as the blind spot size in proportion to the number of pixels of the measured spatial diffusion radius. Image restoration method.

4. In Paragraph 1, The above blindspot neural network includes a diluted convolutional layer, and The expansion rate of the above diluted convolutional layer is set based on the determined blind spot size. Image restoration method.

5. In Paragraph 1, The step of training a blind spot neural network configured according to the above-determined blind spot size in a self-supervised manner using the above-determined single dataset A step of extracting a patch having a preset pixel size from a raw image included in the single dataset above; A step of augmenting training data by applying a geometric transformation to the above patch; and A step of self-updating parameters by backpropagating the loss in a direction that minimizes the Mean Squared Error (MSE) between the prediction result tensor of the above blindspot neural network and the raw image assigned as the inference target. Image restoration method.

6. In Paragraph 1, The raw images included in the single dataset above are provided as training inputs for the blindspot neural network and as inference targets for calculating the loss function. Image restoration method.

7. Data storage for storing one or more instructions; and It includes a processor that executes the above instructions, The above processor Acquire a single dataset containing multiple raw images including noise, and Noise statistics are estimated by performing spatial correlation analysis between raw images included in the single dataset above, and Based on the above noise statistics, the blind spot size is determined, and A blindspot neural network configured according to the blindspot size determined above is trained in a self-supervised manner using the single dataset, and A structure configured to generate a reconstructed image from the raw image using the above blindspot neural network Image restoration device.

8. In Paragraph 7, The above processor In estimating noise statistics by performing spatial correlation analysis between raw images included in the single dataset above, A raw image tensor selected from the single dataset is shifted pixel by pixel in a first direction and a second direction, respectively, a correlation value between the shifted tensor and the original tensor is calculated, and a two-dimensional spatial correlation heatmap is generated using the correlation value. Image restoration device.

9. In Paragraph 8, The above processor In determining the blind spot size based on the above noise statistics, A method configured to measure a spatial diffusion radius in which a correlation value greater than or equal to a preset threshold is maintained in the above 2D spatial correlation heatmap, and to determine a physical blocking area in the form of N×N pixels (where N is an odd number greater than or equal to 1) as the blind spot size in proportion to the number of pixels of the measured spatial diffusion radius. Image restoration device.

10. In Paragraph 7, The above blindspot neural network includes a diluted convolutional layer, and The expansion rate of the above diluted convolutional layer is set based on the determined blind spot size. Image restoration device.

11. In Paragraph 7, The above processor In training a blind spot neural network configured according to the above-determined blind spot size in a self-supervised manner using the above-determined single dataset, A method configured to extract a patch having a preset pixel size from a raw image included in the single dataset, augment training data by applying a geometric transformation to the patch, and self-update parameters by backpropagating the loss in a direction that minimizes the Mean Squared Error (MSE) between the prediction result tensor of the blindspot neural network and the raw image assigned as the inference target. Image restoration device.

12. In Paragraph 7, The raw images included in the single dataset above are provided as training inputs for the blindspot neural network and as inference targets for calculating the loss function. Image restoration device.

Images