Modality-Specific Expert Subnetworks for Medical Image Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training multi-modal foundational models for medical images is challenging due to data heterogeneity and the high cost of acquiring medical images with multiple scanners, leading to entangled representations and reduced representation power when self-supervised learning methods are used without pairs of medical images.
Innovation Solution
The use of modality-specific and cross-modality expert subnetworks within a machine learning-based model allows for the selection and processing of input medical images, merging results from modality-specific subnetworks and then integrating these results using cross-modality subnetworks to perform medical imaging analysis tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If self-supervised learning methods are used without pairs of medical images, then training can proceed without requiring costly multi-modal data collection, but the representations become entangled across modalities and representation power is reduced
Solution Approach 1:
The model is segmented into modality-specific encoder towers (one for each imaging modality) and a shared task-specific head. Each encoder tower processes a specific modality independently, preventing entanglement while allowing the shared head to learn unified task representations. This segmentation resolves the contradiction by maintaining separate processing paths for each modality while still enabling effective multi-modal learning.
Solution Approach 2:
A modality-agnostic intermediate representation layer is introduced between the modality-specific encoders and the task-specific heads. This intermediate layer acts as a mediator that receives processed features from different modalities, aligns them in a shared feature space, and then passes them to the task-specific heads. This intermediary structure enables effective cross-modality information exchange without forcing early entanglement.
2Adaptability or versatility
If contrastive learning methods are used for self-supervised learning, then learning can occur without negative anchors in cross-modality, but the representations become significantly distinct for in-modality while being less distinct for cross-modality
Solution Approach 1:
The model applies different quality characteristics to different parts of the representation pipeline: modality-specific local features are preserved in the encoder towers while global cross-modality alignment is achieved in the intermediate representation layer. This local quality differentiation allows each modality to maintain its distinctive characteristics while also being effectively integrated for cross-modality tasks.
Solution Approach 2:
The model transitions from modality-specific feature spaces to a unified modality-agnostic feature space through the intermediate representation layer. This dimensional transformation allows features from different modalities to be compared and integrated effectively, resolving the issue of representations being too distinct across modalities while maintaining local modality characteristics.
3Device complexity
If a single encoder is used to process all images without modality-specific processing, then the model structure is simpler, but entangled representations across modalities result and representation power is reduced
Solution Approach 1:
The encoder is segmented into multiple modality-specific encoder towers, each dedicated to processing a specific imaging modality. This segmentation prevents entanglement of modality-specific features while maintaining a relatively simple overall structure through the use of shared task-specific heads and a unified intermediate representation layer.
Solution Approach 2:
The model employs universal task-specific heads that can process features from any modality through the shared intermediate representation layer. This multi-functionality allows the model to handle various tasks (classification, segmentation, detection) across different modalities without requiring separate processing paths for each task, balancing complexity and representation power.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for performing a medical imaging analysis task using a machine learning based model (e.g., a foundational model) are provided. One or more input medical images are received. One or more modality-specific expert subnetworks of the machine learning based model are selected for performing a first processing of the one or more input medical images. The first processing of the one or more input medical images is performed for performing a medical imaging analysis task using the one or more selected modality-specific expert subnetworks. Results of the first processing are merged into one or more sets of merged results. One or more cross-modality expert subnetworks of the machine learning based model are selected for performing a second processing of the one or more sets of merged results. The second processing of the one or more sets of merged results is performed for performing the medical imaging analysis task using the one or more selected cross-modality expert subnetworks. Results of the second processing are output.