Cloud-Trained Deep Learning for Microscopic Image Reconstruction

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Current scanning microscopy techniques are slow and impractical for large-scale imaging, and irradiating radiation-sensitive specimens with energetic beams can cause damage, leading to undesirable decreases in signal-to-noise ratio when attempting to mitigate this through reduced intensity or increased scan speed.

Innovation Solution

A method involving training a network in the cloud for microscopic image reconstruction and segmentation using deep convolutional neural networks, which can be pre-trained and adapted for local microscopic systems, allowing for efficient image processing and minimizing data transport and processing time while maintaining high quality.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Area of stationary object

If scanning microscopy techniques are used for large-scale imaging, then imaging area is increased, but imaging speed decreases and specimen damage increases

Engineering Contradiction:
Improveimaging areaVSAvoidimaging speed
Core Design Contradiction:
Area of stationary objectVSSpeed

Solution Approach 1:

The patent divides the large-scale imaging task into multiple smaller image acquisitions that are subsequently reconstructed using deep learning algorithms. The imaging area is segmented into multiple fields of view that are captured sequentially at high speed, then computationally reassembled to create a large-scale image without requiring slow point-by-point scanning across the entire area.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary image reconstruction and enhancement using trained deep learning networks before final image assembly. The network is pre-trained on large datasets to learn reconstruction patterns, allowing rapid processing of acquired images while maintaining high quality output, thus speeding up the overall imaging workflow.

Inventive Principle:
Principle #10Preliminary action

2Speed

If scan speed is increased to mitigate specimen damage, then imaging speed is improved, but signal-to-noise ratio decreases

Engineering Contradiction:
Improvescan speedVSAvoidsignal-to-noise ratio
Core Design Contradiction:
SpeedVSMeasurement precision

Solution Approach 1:

The patent introduces a deep learning reconstruction network as an intermediary between the raw acquired images and the final output. This network learns to reconstruct high-quality images from low-signal input data, effectively mediating between the need for fast scanning and the requirement for high signal-to-noise ratio without requiring slow scanning speeds.

Inventive Principle:
Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent changes the processing parameters through learned transformations in the deep learning network. The network learns optimal parameter transformations during training that enhance signal-to-noise ratio in the reconstructed images while maintaining the fast scan speeds used during data acquisition, effectively decoupling scan speed from output quality.

Inventive Principle:
Principle #35Parameter changes

3Measurement precision

If deep learning networks are trained locally for each microscopic system, then image processing quality is improved, but device complexity and training time increase

Engineering Contradiction:
Improveimage processing qualityVSAvoidsystem complexity
Core Design Contradiction:
Measurement precisionVSDevice complexity

Solution Approach 1:

The patent creates a universal deep learning network that can be deployed across multiple microscopic systems. The network is trained on diverse data from various microscope types and configurations, making it universally applicable. This single trained network serves multiple systems simultaneously, eliminating the need for each system to have its own separately trained network, thus reducing overall complexity while maintaining high processing quality.

Inventive Principle:
Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent creates a master trained network model that can be copied and deployed to multiple microscopic systems. Rather than training separate networks for each system, the trained weights and parameters are replicated across different devices, providing consistent high-quality image processing without repeating the computationally intensive training process at each location.

Inventive Principle:
Principle #26Copying

4Power

If cloud computing is used for network training, then processing power is improved, but data transport time increases

Engineering Contradiction:
Improveprocessing powerVSAvoiddata transport time
Core Design Contradiction:
PowerVSLoss of time

Solution Approach 1:

The patent extracts and separates the training phase from the inference phase. Large datasets are transferred to the cloud only once for the computationally intensive training process, while the resulting trained network model (which is much smaller) is then deployed locally for rapid inference. This extraction of the training function to the cloud maximizes processing power utilization without requiring continuous data transport.

Inventive Principle:
Principle #2Taking out (Extraction)

Solution Approach 2:

The patent performs the computationally intensive network training in advance using cloud computing resources. The trained network model is then stored and reused for multiple inference tasks. This preliminary action of training once in the cloud eliminates the need for repeated data transfers, as the same trained model can process numerous images locally without further cloud communication.

Inventive Principle:
Principle #10Preliminary action

Data Source

PatentUS11482400B2Method, device and system for remote deep learning for microscopic image reconstruction and segmentation
Publication Date: 2022.10.25 FEI CO
  • US11482400B2 patent drawing
  • US11482400B2 patent drawing
  • US11482400B2 patent drawing

AI summary

The present invention relates to a method of training a network for reconstructing and/or segmenting microscopic images comprising the step of training the network in the cloud. Further, for training the network in the cloud training data comprising microscopic images can be uploaded into the cloud and a network is trained by the microscopic images. Moreover, for training the network the network can be benchmarked after the reconstructing and/or segmenting of the microscopic images. Wherein for benchmarking the network the quality of the image(s) having undergone the reconstructing and/or segmenting by the network can be compared with the quality of the image(s) having undergone reconstructing and/or segmenting by already known algorithm and/or a second network.