Learnable Visual Prompt Engineering for Medical Image Adaptation

Solution Overview

Problem

Existing techniques for deploying machine learning models on medical images from new domains are excessively time-consuming and computationally expensive due to the need for retraining or fine-tuning the models.

Innovation Solution

Implementing a pre-processing image transformation with iteratively learned parameters that adapts medical images to match the visually stylistic characteristics of the original domain, allowing the pre-trained machine learning model to perform accurately without retraining.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If retraining or fine-tuning is performed on annotated medical images from the new domain, then the machine learning model can accurately perform visual inferencing tasks on out-of-domain medical images, but the process becomes excessively time-consuming and computationally expensive

Engineering Contradiction:

Improveaccuracy of visual inferencing taskVSAvoidtime for retraining or fine-tuning

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the adaptation process into two independent parts: (1) a pre-processing transformation module that modifies input images, and (2) a frozen pre-trained model that performs inference. This segmentation allows the pre-trained model to remain unchanged while adapting to new domains through image transformation alone, eliminating the need for time-consuming retraining or fine-tuning of the entire model.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies pre-processing transformations to medical images before they are fed into the pre-trained model. This preliminary action of transforming images to match the visual characteristics of the original training domain enables the model to process out-of-domain images effectively without requiring any modification or retraining, thus saving significant time and computational resources.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If retraining or fine-tuning is performed on annotated medical images from the new domain, then the machine learning model can accurately perform visual inferencing tasks on out-of-domain medical images, but the computational cost increases excessively

Engineering Contradiction:

Improveaccuracy of visual inferencing taskVSAvoidcomputational cost

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent segments the adaptation process into two independent parts: (1) a pre-processing transformation module that modifies input images, and (2) a frozen pre-trained model that performs inference. This segmentation allows the pre-trained model to remain unchanged while adapting to new domains through image transformation alone, eliminating the need for time-consuming retraining or fine-tuning of the entire model.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent uses a lightweight pre-processing transformation module with learnable parameters that requires minimal computational resources compared to retraining the entire pre-trained model. This disposable-like approach of using a simple transformation layer instead of full model retraining significantly reduces computational cost while maintaining effectiveness.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

3Adaptability or versatility

If the pre-processing transformation uses iteratively learned parameters, then the transformation can adapt to match the visually stylistic characteristics of the original domain, but the transformation complexity increases

Engineering Contradiction:

Improveability to match visually stylistic characteristicsVSAvoidcomplexity of pre-processing transformation

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent employs a pre-processing transformation with learnable parameters that are iteratively optimized to match the visual characteristics of the original training domain. By changing and optimizing these parameters based on the target domain's visual style, the transformation adapts to out-of-domain images effectively while maintaining a relatively simple architectural structure.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20250149169A1Learnable visual prompt engineering

Publication Date: 2025.05.08 GE PRECISION HEALTHCARE LLC

AI summary

Systems or techniques for facilitating learnable visual prompt engineering are provided. In various embodiments, a system can access a medical image and a pre-trained machine learning model that is configured to perform a diagnostic or prognostic inferencing task. In various aspects, the system can apply a pre-processing transformation to one or more pixels or voxels of the medical image, thereby yielding a transformed version of the medical image, wherein the pre-processing transformation can convert an input pixel or voxel intensity value to an output pixel or voxel intensity value via one or more parameters that are iteratively learned. In various instances, the system can perform the diagnostic or prognostic inferencing task, by executing the pre-trained machine learning model on the transformed version of the medical image.