Multi-Modal 3D Medical Image Registration via Multi-Scale Joint Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning methods for medical image registration, particularly for multi-modal images like magnetic resonance and ultrasound images of the prostate, face challenges in accurately determining deformation fields due to low similarity between image modalities, leading to inaccurate registration and overfitting issues.
Innovation Solution
A method involving a multi-scale self-attention network for simultaneous training of segmentation and registration models, using multi-scale segmentation losses and registration losses to optimize deformation fields, constrained by level set energy function and gradient losses, to enhance registration accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If segmentation is performed before registration to handle low similarity between multi-modal images, then the similarity loss can be built according to target region contour and background interference is removed, but segmentation errors affect subsequent registration performance and cause wrong convergence and overfitting
Solution Approach 1:
The patent merges segmentation and registration tasks into a unified joint training framework. The segmentation network and registration network share the same input images and are trained simultaneously with a combined loss function that includes both segmentation loss and registration loss. This allows the registration network to learn from both the original images and the segmentation results, while the segmentation network benefits from the spatial context provided by registration, thereby reducing the negative impact of segmentation errors on registration performance.
Solution Approach 2:
The patent implements a feedback mechanism where the segmentation results are fed back to the registration network as additional input. The registration network uses both the original images and the segmentation masks to compute the deformation field. This feedback loop allows the system to iteratively refine both segmentation and registration results during joint training, preventing wrong convergence by continuously adjusting based on registration performance.
2Device complexity
If deep learning registration models use encoding and decoding frameworks with deformation parameters output only at the end, then the model structure is simplified, but the registration performance is limited because it does not incorporate the progressive registration process features
Solution Approach 1:
The patent segments the registration process into multiple stages by introducing intermediate deformation fields at different levels of the decoding framework. Instead of outputting only the final deformation field, the model generates intermediate deformation fields at multiple stages of the decoding process. This allows the registration to progress through multiple refinement steps, incorporating progressive registration process features while maintaining a relatively simple encoding-decoding framework structure.
Solution Approach 2:
The patent makes the model structure dynamic by allowing the deformation field to be computed and refined at multiple stages rather than just at the end. The intermediate deformation fields enable the model to adaptively adjust the registration process at different levels of detail, transforming the static end-to-end framework into a dynamic multi-stage process that can incorporate progressive refinement features.
3Ease of manufacture
If unsupervised methods are used for medical image registration to avoid high label acquisition costs, then the method can be applied to actual tasks, but the similarity between magnetic resonance and ultrasound images is low making it difficult to train registration models
Solution Approach 1:
The patent introduces segmentation results as an intermediary that bridges the gap between multi-modal images with low similarity. The segmentation networks first process the magnetic resonance and ultrasound images separately to generate modality-specific segmentation masks. These segmentation results serve as intermediaries that capture the target region structures, which are then used to guide the registration process. This intermediary approach allows the registration model to train effectively despite the low direct similarity between the original multi-modal images.
Solution Approach 2:
The patent changes the parameters used for training by introducing a joint loss function that combines segmentation loss and registration loss. This composite loss function allows the model to optimize both segmentation accuracy and registration accuracy simultaneously. By changing the training objective to include multiple loss components, the model can learn effective representations for both tasks, overcoming the low image similarity problem while maintaining unsupervised training benefits.
Data Source
AI summary
The present invention belongs to the field of medical image registration, and more specifically, relates to a building method of a multi-modal three-dimensional medical image segmentation and registration model, and an application thereof. The method includes: acquiring medical images of two modalities of each target, and respectively using the images as a reference image and a floating image to acquire a training sample; and using the training sample to simultaneously optimize three network parameters, so as to acquire a segmentation and registration model formed by a reference image segmentation model, a floating image segmentation model, and a registration model. The reference image and floating image segmentation models are respectively used to perform multi-scale segmentation on corresponding images to acquire multi-scale segmentation results having the same maximum scale as the original images. The registration model are used to acquire a multi-scale deformation field on the basis of the reference image, the floating image, a maximum scale reference image segmentation result, and a maximum scale floating image segmentation result. Each of a segmentation loss and a registration loss are the sum of segmentation and registration losses in each scale, and the segmentation loss includes a first-order gradient loss and/or a level set energy function loss. The present invention can improve registration accuracy.