Single-Shot Multi-Label Medical Image Segmentation via Vision Transformers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for localization and region-of-interest segmentation in medical imaging, particularly using foundation models, are laborious, repetitive, and prone to clinician fatigue, leading to inaccuracies and increased costs, while previous attempts to automate these tasks with grounding foundation models have not been successful.
Innovation Solution
A computer-implemented method utilizing a trained vision transformer model and a contrastive similarity metric learning model to automatically determine pixel-level feature similarities, enabling data adaptive single-shot multi-label segmentation by labeling pixels in medical images with initial segmentation masks, and refining these masks using a promptable segmentation model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual localization and segmentation methods are used, then clinicians can identify regions of interest with detailed control, but the process becomes laborious and repetitive leading to clinician fatigue and reduced accuracy
Solution Approach 1:
The system enables automated localization and segmentation where the computational model performs the localization task independently by processing medical images through trained vision transformer and contrastive similarity metric learning models, eliminating the need for manual clinician intervention in repetitive localization tasks while maintaining accuracy through data-driven feature matching
Solution Approach 2:
The patent replaces manual mechanical localization operations with an automated computational system that uses deep learning models (vision transformer, contrastive metric learning) to automatically identify and segment regions of interest, substituting human manual tracing and annotation with algorithm-based pixel-level feature comparison and segmentation mask generation
2Productivity
If foundation models are used for automated localization, then processing time is reduced, but previous attempts have not been successful in achieving accurate segmentation
Solution Approach 1:
The system changes the parameter space by transforming medical images into feature vectors through a vision transformer model, then applying contrastive similarity metric learning to learn optimal feature space transformations that maximize discrimination between different tissue types and structures, enabling accurate automated segmentation without manual intervention
Solution Approach 2:
The patent introduces an intermediary contrastive similarity metric learning model that bridges the gap between raw image data and final segmentation masks, using learned feature representations to mediate the matching process between template images and target medical images, thereby achieving both speed and accuracy in automated localization
3Measurement precision
If multiple regions of interest are segmented separately, then each region can be analyzed in detail, but the process becomes repetitive and increases costs
Solution Approach 1:
The system implements a universal segmentation framework where a single trained model can simultaneously segment multiple different types of regions of interest (e.g., different anatomical structures, lesions, or tissues) within the same medical image by using template images for each region type and applying the same contrastive metric learning approach to match all regions in one processing pass
Data Source
AI summary
A method includes obtaining a medical image and receiving a selection of both a template image and a region of interest within the template image. The method includes inputting both the medical image and the template image into a trained vision transformer model and outputting from the trained vision transformer model both pixel level feature vectors from the medical image and a reference pixel level feature vector from the region of interest of the template image. The method includes inputting both the pixel level feature vectors and the reference pixel level feature vector into a trained contrastive similarity metric learning model and outputting from the trained contrastive similarity metric learning model pixel that are similar to reference pixels. The method includes labeling the pixels in the medical image with a segmentation mask, wherein the pixels that are labeled in the medical image correspond to the region of interest.