Multimodal Image Reconstruction Using Physics-Informed Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing non-destructive testing methods for semiconductor packaging, such as X-ray CT and scanning acoustic microscopy, face challenges with noise, reduced resolution, and insufficient contrast, particularly in detecting micro-scale defects in complex integrated circuits due to scattering effects and material overlap.
Innovation Solution
A multimodal image fusion approach combined with physics-informed neural networks (PINNs) is used to enhance image quality by integrating physical principles, such as X-ray attenuation and acoustic wave propagation, to refine image reconstruction and improve defect detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If X-ray CT imaging is used to detect internal defects, then internal defect detection capability is improved, but image quality deteriorates due to scattering effects from high atomic number elements
Solution Approach 1:
The patent introduces a deep learning-based image reconstruction system as an intermediary between raw X-ray data and final images. This system uses neural networks to model the complex scattering processes and generate corrected images that compensate for the degrading scattering effects, thereby maintaining both internal defect detection capability and image quality.
Solution Approach 2:
The patent transforms the imaging approach by changing from direct image acquisition to parameter-based reconstruction. Instead of relying on direct X-ray image intensity, the system uses deep learning models to reconstruct images from raw data, effectively changing the parameters used for image generation to overcome scattering limitations.
2Difficulty of detecting and measuring
If scanning acoustic microscopy is used to detect subsurface flaws, then subsurface flaw detection is improved, but image quality deteriorates due to high noise levels and reduced resolution
Solution Approach 1:
The patent employs a deep learning reconstruction system as an intermediary that processes noisy SAM data. The neural network model acts as a mediator that separates signal from noise, reconstructing clear images from the noisy input data while preserving subsurface flaw detection capabilities.
Solution Approach 2:
The system creates a computational copy of the imaging process using deep learning models. Instead of directly analyzing noisy physical measurements, the system learns the mapping from raw data to clean images through training, effectively copying the ideal imaging process to overcome noise and resolution limitations.
3Measurement precision
If multi-modal spectral fusion is performed to enhance image quality, then resolution and contrast are improved, but computational complexity increases
Solution Approach 1:
The patent replaces traditional computational imaging methods with deep learning-based approaches. Instead of using complex iterative algorithms for image fusion and reconstruction, the system uses trained neural networks that perform these operations more efficiently, reducing computational complexity while maintaining or improving resolution and contrast.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The method provides clearer and more accurate images with enhanced resolution and defect visibility, reducing computational demands and overcoming limitations of traditional methods.
Implementation Method 1
a physics-informed loss function associated with one or more physical principles associated with X-ray attenuation or scattering
Implementation Method 2
a physics-informed loss function associated with one or more physical principles associated with X-ray attenuation or scattering
Implementation Method 3
the multi-modal imaging system is configured to provide the plurality of images via optical acquisition and X-ray acquisition
Implementation Method 4
the multi-modal imaging system is configured to provide the plurality of images via optical acquisition and X-ray acquisition
Implementation Method 5
performing multi-modal spectral fusion of the plurality of images
Implementation Method 6
decomposing the plurality of images into one or more corresponding low-frequency components (LFCs) and one or more corresponding high-frequency components (HFCs)
Implementation Method 7
a physics-informed loss function associated with (i) a density of an integrated circuit advanced packaging material, (ii) a wave propagation speed, and (iii) one or more second-order derivatives that corresponds to a wavefront with respect to time and spatial coordinates
Data Source
AI summary
A method comprising receiving a plurality of images from a multi-modal imaging system; generating a plurality of filtered measurements by performing multi-modal spectral fusion of the plurality of images; and generating, using a physics-informed neural network (PINN) trained based on one or more physical principles associated with X-ray attenuation or scattering, a reconstructed object image based on the plurality of filtered measurements, wherein generating the reconstructed object image comprises (i) generating, using the PINN, a system matrix for an X-ray imaging forward model by refining one or more coefficients of the system matrix based on a physics-informed loss function, and (ii) generating, using the X-ray imaging forward model and based on the plurality of filtered measurements, the reconstructed object image.