3D Anatomical Reconstruction from 2D Images via Neural Network Warping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current 3D imaging technologies, such as CT scans, are limited by high radiation dosage, high cost, immobility, and inability to accommodate versatile patient positions, making them unsuitable for certain diagnostic and surgical planning applications. Additionally, standard approaches for training neural networks to map 2D images to 3D point clouds face challenges due to the lack of consistent representation in 3D polygon meshes, leading to incorrect results.
Innovation Solution
A method is developed to train a neural network for reconstructing 3D point clouds from 2D images by selecting a template point cloud, non-rigidly registering it with other point clouds to create consistent warped templates, and using these templates along with corresponding 2D images to map 2D anatomical images into 3D point clouds, enabling the reconstruction of 3D anatomical structures from 2D images like x-rays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If CT scan is used for 3D imaging, then imaging accuracy and resolution are improved, but radiation dosage and cost increase
Solution Approach 1:
The patent uses 2D X-ray images as a copy or alternative representation of the 3D anatomical structure. Instead of directly acquiring 3D data through CT scans, the system reconstructs 3D point clouds from 2D X-ray images, thereby obtaining imaging information with significantly reduced radiation exposure while maintaining diagnostic accuracy.
Solution Approach 2:
The patent transforms the imaging approach from 3D direct acquisition (CT) to 2D projection (X-ray) and back to 3D reconstruction through neural networks. This dimensional transformation allows the system to achieve 3D imaging capabilities using 2D images, reducing radiation dosage while preserving spatial information.
2Measurement precision
If CT scanner is used, then 3D imaging capability is provided, but device mobility and patient position flexibility are reduced
Solution Approach 1:
The system creates a digital 3D copy of the anatomical structure from 2D X-ray images, eliminating the need for a physical 3D imaging device during patient examination. This allows the same 3D reconstruction capability to be applied to patients in various positions and orientations without requiring movement of heavy imaging equipment.
Solution Approach 2:
By converting 2D X-ray images into 3D point cloud representations through neural network reconstruction, the system enables 3D imaging capability to be applied to patients in diverse positions and orientations, overcoming the mobility limitations of traditional CT scanners.
3Ease of manufacture
If standard neural network training methods are used for 2D to 3D mapping, then training process is simple, but reconstruction accuracy deteriorates due to inconsistent 3D representation
Solution Approach 1:
The patent changes the parameter representation of 3D structures from polygon meshes to point clouds with consistent coordinate ordering. This parameter transformation enables neural networks to learn consistent mappings from 2D images to 3D structures, significantly improving reconstruction accuracy while maintaining training simplicity through the use of standardized point cloud formats.
Solution Approach 2:
The system applies different processing approaches to different aspects of the 3D representation. By using point clouds with consistent coordinate ordering rather than traditional polygon meshes, the patent ensures that local geometric information is preserved consistently across the entire 3D structure, improving overall reconstruction accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
There is provided a method of training a neural network for reconstructing of a 3D point cloud from 2D image(s), comprising: extracting (104) point clouds each represented by an ordered list of coordinates, from 3D anatomical images depicting a target anatomical structure, selecting (106) one of the plurality of point clouds as a template, non-rigidly registering the template with each of the point clouds to compute (108) a respective warped template having a shape of the respective point cloud and retaining the coordinate order of the template, wherein the warped templates are consistent in terms of coordinate order, receiving (110) 2D anatomical images depicting the target anatomical structure depicted in corresponding 3D anatomical images, and training (114) a neural network, according to a training dataset of the warped templates and corresponding 2D images, for mapping 2D anatomical image(s) into a 3D point cloud.