Medical Image Foundation Models With Modality Expert Subnetworks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training multi-modal foundational models for medical images is challenging due to data heterogeneity and high costs associated with imaging patients using multiple scanners, leading to entangled representations and reduced representation power without sufficient pairs of medical images.
Innovation Solution
Utilizing modality-specific and cross-modality expert subnetworks within a machine learning model, where modality-specific subnetworks process individual image types and cross-modality subnetworks integrate features across different modalities, enabling disentangled representations and efficient cross-modality information exchange.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If all images are processed by the same encoder without pairs of medical images, then the training can proceed without requiring paired data from different modalities, but the representations become entangled across modalities and the representation power is reduced
Solution Approach 1:
The encoder is segmented into multiple modality-specific encoders, each dedicated to processing a specific medical image modality. This segmentation allows each encoder to learn modality-specific features independently, preventing entanglement while maintaining the ability to train without paired data across modalities.
Solution Approach 2:
The system employs a universal framework that can process multiple modalities through a shared architecture of modality-specific encoders and a cross-modality integrator. This multi-functional design enables the system to handle different modalities independently while still achieving cross-modality representation when needed.
2Loss of information
If pairs of medical images from different modalities are collected for training, then the representation power can be improved, but the costs associated with imaging the same patients with multiple scanners become prohibitively high
Solution Approach 1:
The system uses self-supervised learning where each modality serves as its own source of training signals. The modality-specific encoders are trained independently on single-modality data using self-supervised objectives, eliminating the need for expensive paired multi-modality data while still achieving effective representation learning.
3Ease of manufacture
If contrastive learning methods are used for self-supervised learning, then the training can proceed without labels, but the representations become significantly distinct for in-modality while being less distinct for cross-modality
Solution Approach 1:
A cross-modality integrator is introduced as an intermediary component that receives features from multiple modality-specific encoders and integrates them into unified cross-modality representations. This mediator ensures that cross-modality representations are distinct and meaningful, addressing the limitation of standard contrastive learning methods.
Data Source
AI summary
Systems and methods for performing a medical imaging analysis task using a machine learning based model (e.g., a foundational model) are provided. One or more input medical images are received. One or more modality-specific expert subnetworks of the machine learning based model are selected for performing a first processing of the one or more input medical images. The first processing of the one or more input medical images is performed for performing a medical imaging analysis task using the one or more selected modality-specific expert subnetworks. Results of the first processing are merged into one or more sets of merged results. One or more cross-modality expert subnetworks of the machine learning based model are selected for performing a second processing of the one or more sets of merged results. The second processing of the one or more sets of merged results is performed for performing the medical imaging analysis task using the one or more selected cross-modality expert subnetworks. Results of the second processing are output.