Electron Microscopy Image Restoration Using Adversarial Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electron microscopy images are limited by various stochastic and deterministic image distortions, such as X and Y jitter, fast scanning artefacts, Poisson noise, readout noise, dark current noise, X-ray noise, and image saturation, which reduce the signal-to-noise ratio (SNR) and limit the extraction of quantitative information, especially in beam-sensitive materials and dynamic imaging.
Innovation Solution
A method for training an artificial neural network using adversarial training to reduce noise and artefacts, utilizing a generator network to output undistorted images from distorted inputs, with a discriminator network to differentiate between synthetic and restored images, covering a wide range of noise and artefact types across different electron microscopy modalities and resolutions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the electron dose is increased to improve the signal-to-noise ratio, then the image quality is improved, but the cumulative electron dose to the sample increases which reduces applicability for beam-sensitive materials
Solution Approach 1:
The method performs preliminary actions by acquiring multiple individual frames with controlled electron doses and performing image registration before averaging. This preliminary processing allows the system to achieve improved signal-to-noise ratio through computational averaging while maintaining lower cumulative electron dose per frame, making the technique applicable to beam-sensitive materials.
2Measurement precision
If beam current or acquisition time is increased to improve the signal-to-noise ratio, then the image quality is improved, but other types of image distortions such as shear or rotation increase
Solution Approach 1:
The method performs preliminary image registration to correct distortions such as shear and rotation before averaging the frames. This preliminary correction allows the system to maintain shorter acquisition times and lower beam currents while still achieving high signal-to-noise ratio, thereby avoiding the accumulation of time-dependent distortions.
Solution Approach 2:
The image registration process provides feedback by identifying and correcting positional and orientational deviations between frames. This feedback mechanism enables the system to compensate for distortions that would otherwise accumulate with increased acquisition time or beam current, maintaining image precision while improving signal-to-noise ratio.
3Measurement precision
If a large number of individual frames are acquired to improve the signal-to-noise ratio through averaging, then the image quality is improved, but the cumulative electron dose to the sample increases
Solution Approach 1:
The method performs preliminary image registration to align frames before averaging, ensuring that the improvement in signal-to-noise ratio is achieved efficiently. By correcting misalignments beforehand, the system can use fewer frames for averaging, thereby reducing the cumulative electron dose while still achieving the desired image quality.
Data Source
AI summary
In a method, for training an artificial neural network (ANN) to reduce noise and/or artefacts in an electron microscopy image, a plurality of training image pairs is generated. For each pair, an undistorted synthetic specimen image and a distorted image are created by simulating additional noise and/or artefact features. The ANN is trained, in which the distorted images are used as input and the corresponding undistorted images as output. An adversarial training strategy is used in which the ANN is trained, as a generator network, in conjunction with concomitantly training a further ANN, as a discriminator network, to differentiate output produced by the generator network from synthetic images in the training set. In training, parameters of the ANN and further ANN are optimized using a generator loss function and a discriminator loss function, in which a dependency exists between said loss functions to train the networks in an adversarial manner.