Performance evaluation method and system for multimodal medical images
Patent Information
- Application Number
- PCT/CN2025/106754
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-07-03
- Publication Date
- 2026-10-22
Smart Images
Figure CN2025106754_22102026_PF_FP_ABST
Abstract
Description
Performance Evaluation Methods and Systems for Multimodal Medical Images Technical Field
[0001] This invention relates to the field of medical image analysis technology. Background Technology
[0002] In medical imaging, the auxiliary diagnosis of a particular disease often involves many different types / modalities / sequences of imaging methods. For example, in the diagnosis of breast diseases, the classification of ductal carcinoma in situ, invasive ductal carcinoma, lobular carcinoma, and non-malignant tumors (fibroadenoma, adenosis, hyperplasia, etc.) includes multimodal / multi-sequence image categories such as mammography, breast MRI T2-weighted, T1 plain scan, contrast-enhanced (DCE), DWI, and ADC. Another example is the diagnosis of gliomas, which can employ various imaging techniques such as T1, T2W (especially T2 Flair), DWI, DWI perfusion imaging, DSC imaging, and arterial spin labeling imaging. Similar approaches are used for the auxiliary diagnosis and postoperative evaluation of diseases such as prostate cancer, rectal cancer, and lung cancer.
[0003] In existing technologies, for the auxiliary diagnosis of a certain type of disease, deep learning is used to perform auxiliary diagnostic analysis on images of certain pre-selected modalities. For example, the auxiliary diagnosis of breast cancer usually uses DCE and T2W or DWI as the objects for processing, while the segmentation of glioma usually uses T2 Flair as the main method, combined with T1 or other modalities for image fusion or feature fusion.
[0004] However, current technologies lack consideration for the following questions: 1) Why select these modal image combinations instead of others? (The answer may be clinical experience and subjective judgment.) 2) When medical conditions limit the availability of certain modalities, can other modalities be used instead, and how significant is the difference in effectiveness? For example, dynamic contrast-enhanced CT (DCE) requires the injection of contrast agents, which is expensive, and some patients are allergic to contrast agents and cannot receive the injection; how can a balance be struck between treatment effectiveness and cost? 3) In the process of multimodal analysis (whether it is diagnosis by radiologists or computer-aided diagnosis), if the current modal combination cannot accurately assess the disease classification (benign / malignant), which modalities / sequences of images should be added to most effectively improve diagnostic accuracy? Summary of the Invention
[0005] To address the shortcomings of the aforementioned technologies, this invention provides a performance evaluation method for multimodal medical images, solving the technical problem of the lack of a more objective reference standard when combining single-modal images in existing technologies.
[0006] To address the aforementioned technical problems, this invention provides a method for performance evaluation of multimodal medical images, comprising the following steps:
[0007] A case sample set for the same disease is obtained, and a multimodal medical image sequence corresponding to each case sample in the case sample set is obtained. The multimodal medical image sequence contains... A single-modal medical image sequence;
[0008] A portion of the case samples were selected as the training set, and the remaining case samples were selected as the test set. Both the training set and the test set were assigned the same case-level classification label.
[0009] Extract from the multimodal medical image sequences corresponding to the training set A single-modal feature vector;
[0010] The By combining the eigenvectors of the single modality, we obtain A combination of feature vectors;
[0011] A classification model is trained for each combination of feature vectors, so that each classification model has the ability to identify the same disease through the corresponding combination of feature vectors.
[0012] The test set is used to evaluate the classification performance of each classification model, which is used as the classification performance of each feature vector combination for the same disease, thereby obtaining the classification performance of multimodal medical images formed by different combinations of unimodal medical images for the same disease.
[0013] Furthermore, the trained feature extractor is used to automatically extract single-modality feature vectors; the training steps of the feature extractor include:
[0014] Positive sampling is performed on each unimodal medical image sequence in the multimodal medical image sequence in the training set: the unimodal medical image sequence is examined, and images containing lesions are selected as sampling images;
[0015] A corresponding disease label is set for each lesion on each sampled image, and the sampled images with disease labels are used to build a training dataset.
[0016] ImageNet1000 is used to pre-train the CNN neural network. The network weights of the CNN neural network are retained, and the last softmax activation layer is replaced with a softmax activation layer that satisfies the number of disease label categories, thereby constructing a feature extractor.
[0017] The feature extractor is trained using the training dataset.
[0018] Furthermore, the extraction process of single-modal feature vectors includes the following steps:
[0019] The number of slices in each monomodal medical image sequence is unified to S by sampling or interpolation, and the scanning positions of the corresponding slices are basically consistent.
[0020] The single-modal medical image sequence is input into the trained feature extractor, and the output of the last fully connected layer is extracted as the feature vector.
[0021] This invention also provides a performance evaluation system for multimodal medical images, including a multimodal medical image sequence input module, N parallel single-modal feature extraction channels, a feature combination module, and... A classification model and a classification performance evaluation module;
[0022] The multimodal medical image sequence input module is used to input the following data: Multimodal medical image sequences of various single-modal medical image sequences are input into N parallel single-modal feature extraction channels respectively;
[0023] The single-modal feature extraction channel includes an image preprocessing module and a feature extractor. The feature extractor is obtained by training a CNN neural network using a training set and has the ability to extract single-modal features.
[0024] The feature combination module is used to combine single-modal feature vectors and generate the resulting... Each feature vector combination is input to... In each classification model;
[0025] The classification model is used to identify a disease type based on a combination of corresponding feature vectors;
[0026] The classification performance evaluation module is used to evaluate the classification performance of the classification model based on the classification results of the classification model on the test set, and to evaluate the classification performance of each feature vector combination for the same disease, thereby obtaining the classification performance of multimodal medical images formed by different combinations of single-modal medical images for the same disease.
[0027] Compared with the prior art, the beneficial effects of the present invention include:
[0028] 1. First, this invention does not focus on auxiliary disease classification and prediction, or lesion location detection. Instead, it focuses on retrospective study, which solves the problem of multimodal analysis mentioned above: which monomodal medical images should be selected for combination to achieve a more objective, accurate, and realistic result.
[0029] 2. This invention is end-to-end; only images and classification labels are needed as input during model training and multimodal evaluation. Features are extracted using a feature extractor, requiring no further manual intervention. Image-level classification labels are needed during training, while only case-level classification labels are needed during evaluation. This significantly reduces the time required for multimodal evaluation.
[0030] 3. The evaluation process for the invention is longitudinal. Longitudinal evaluation means that the clinical diagnostic efficacy evaluation of a multimodal combination is compared only with the efficacy of other combinations, not with the accuracy of other algorithms or physician diagnostic methods. Since the workflow for evaluation is the same, this ensures the objectivity of the evaluation process and improves the reliability of the results. Attached Figure Description
[0031] Figure 1 is a flowchart of the performance evaluation method for multimodal medical images in this specific embodiment;
[0032] Figure 2 shows the structure of the Transformer model. Detailed Implementation
[0033] The present invention will now be described in detail with reference to the accompanying drawings.
[0034] This invention provides a performance evaluation method for multimodal medical images, as shown in Figure 1 (where the right-angled boxes represent the method flow and the rounded boxes represent the data), including the following steps:
[0035] 1) Constructing samples
[0036] A case sample set for the same disease is obtained, and a multimodal medical image sequence corresponding to each case sample in the case sample set is obtained. The multimodal medical image sequence contains... A single-modal medical image sequence;
[0037] Image preprocessing is performed on the input multimodal / multiparameter image sequence. Preprocessing steps typically include image cropping, enhancement, and denoising. The aim is to make image features, especially lesion features, more clearly distinguishable from background features.
[0038] Multimodal medical image sequences corresponding to a portion of case samples were selected as the training set, and multimodal image sequences corresponding to the remaining case samples were selected as the test set. Both the training set and the test set were assigned the same case-level classification label. The case-level label is generally obtained through pathological in vivo detection and is considered the gold standard.
[0039] 2) Training the feature extractor
[0040] 2D CNN Training. The purpose of training is to generate a feature extractor for this image sequence to extract feature vectors from the images. This step is further divided into three steps: slice sampling, label setting, and classifier training.
[0041] Step 2.1, slice sampling. Since multimodal medical images (MRI / CT) are usually 3D image sequences, and lesions (benign / malignant tumors, etc.) usually do not appear in every image of the sequence, it is necessary to manually examine the 3D sequence and select images containing lesions as positive samples.
[0042] Step 2.2, Label Setting. Different labels (image-level classification labels, only related to the location of lesions on the image) are set for different types of diseases. For example, for multimodal analysis of breast mpMRI, a two-category classification can be defined as non-malignant / malignant tumor, or a three-category classification can be defined as normal breast / benign (including adenosis, fibroadenoma, hyperplasia, and other diseases) / malignant tumor. Further classifications can be made for different tumor subtypes (ductal carcinoma in situ, invasive ductal carcinoma, lobular carcinoma, etc.). Category balance needs to be considered when setting labels.
[0043] ImageNet1000 (a dataset built by Stanford University professor Fei-Fei Li to address overfitting and generalization issues in machine learning, available at https: / / image-net.org / ) was used to pre-train the CNN neural network. The network weights of the CNN neural network were retained, and the last softmax activation layer was replaced with a softmax activation layer that satisfies the number of disease label categories, thereby constructing a feature extractor.
[0044] Step 2.3, Classifier Training. A CNN network is used to train the classifier on the sampled images and labels. Various commonly used classification models can be selected for the CNN model. In our experiments, we chose models such as VGG16 / 19, ResNet50 / 101 / 152, InceptionV3, InceptionResnet2, DenseNet169, and DenseNet201. These models show slight differences in classification accuracy and final evaluation accuracy, but these differences do not significantly affect the final conclusion. (Avoid using EffecientNet or ViT (VisionTransformer) structures, as their training performance is not ideal.) The training process mainly involves pre-training with ImageNet1000, retaining the network weights, replacing the last softmax activation layer with a softmax layer that satisfies the current number of classification categories, and then training with our sampled training dataset.
[0045] 3) Extracting single-modal feature vectors
[0046] Extract from the multimodal medical image sequences corresponding to the training set To extract single-modal feature vectors, a trained feature extractor is used to automatically extract them. First, the images are serialized. Then, each modality image is sampled / interpolated with the same number of slices (here, S). Generally, it is required that the scanning positions of the corresponding slices for each modality are basically consistent, so that the combination of feature vectors at each position can better represent the lesion category. Finally, a trained CNN is used to extract features from each image. Specifically, the image is input into the CNN model, and the output of the last fully connected layer is truncated as the feature vector (in the experiment, we selected the outputs of fully connected layers and convolutional layers for comparison, and the fully connected layer performed better).
[0047] 4) Combining feature vectors
[0048] Permutations and combinations of multimodal feature vectors. For N modal feature vectors, we assume the order of arrangement does not affect the final evaluation result; therefore, we combine the feature vectors. According to the permutation and combination formula, the total number of combinations of N modal feature vectors is: .
[0049] 5) Training the classification model
[0050] A classification model is trained for each combination of feature vectors, so that each classification model has the ability to identify the same disease through the corresponding combination of feature vectors.
[0051] An N-channel parallel Transformer model is employed, and the model details are shown in Figure 2. For the input of a combination of feature vectors from N modalities, a model with dimension [missing information] is designed. The Transformer model was used for classification evaluation (our experiments compared 3D multi-channel CNN, multi-channel LSTM, and the Transformer model, and the Transformer model ultimately achieved the best classification performance). Here, S represents the number of slices, and N represents the number of channels. The final output of each channel / slice of the Transformer is then classified using softmax through pooling and fully connected layers. The Transformer model can be trained using case-level classification labels. A Transformer model needs to be trained for each permutation and combination, so this stage requires a total of [number missing] training sessions. Several Transformer models were used. Finally, these models were evaluated on a test set, and the efficiency of different modality combinations was compared.
[0052] 6) Performance Evaluation
[0053] A test set was used to evaluate the classification performance of each model, providing the classification performance for each feature vector combination on the same disease. This yielded the classification performance of multimodal medical images formed by different combinations of unimodal medical images on the same disease. Classification performance included accuracy and / or recall.
[0054] The following example uses multimodal medical images of breast cancer MRI:
[0055] Research Background and Objectives
[0056] Currently, MRI DCE (dynamic contrast-enhanced imaging) is the most effective (highly sensitive) method for treating breast cancer. However, if it is not feasible to perform DCE imaging due to factors such as high cost or patient intolerance to contrast agents, can deep learning classification combined with multimodal imaging (T1NFS, T2W, DWI, etc.) achieve similar results?
[0057] Experimental Design
[0058] Image data: 595 breast MRI cases, including sequences (8 types in total): T1NFS, T2W, trace weighted b50 / b850, ADC, extrapolated b-value images (BVAL), DCE early / late (reference comparison).
[0059] Tags: Taken from pathology report + imaging report, location (left / right), category (Normal / Benign / Malignant, three-category classification);
[0060] Image preprocessing: DWI denoising, center / left / right cropping, Z-Score regularization;
[0061] 2D image classification models: ResNet50, ResNet101, InceptionV3, InceptionRes2, Dense169, Dense201
[0062] 3D multimodal classification models: Direct Pool model, LSTM+GAP, Transformer;
[0063] Training strategy: Perform 3:1 cross-validation on 277 cases; use the remaining 318 cases as the test set;
[0064] Experimental results
[0065]
[0066] In the table, DCE stands for Dynamic Contrast Enhancement; DWI stands for Diffusion Weighted Imaging.
[0067] 1. Revealed the sequence combinations with / without DCE that achieved the highest accuracy;
[0068] 2. When the lesion cannot be determined, which sequences can be added to improve accuracy (87%~92%).
Claims
1. A performance evaluation method for multimodal medical images, characterized in that, Includes the following steps: A case sample set for the same disease is obtained, and a multimodal medical image sequence corresponding to each case sample in the case sample set is obtained. The multimodal medical image sequence contains... A single-modal medical image sequence; A portion of the case samples were selected as the training set, and the remaining case samples were selected as the test set. Both the training set and the test set were assigned the same case-level classification label. Extract from the multimodal medical image sequences corresponding to the training set A single-modal feature vector; The By combining the eigenvectors of the single modality, we obtain A combination of feature vectors; A classification model is trained for each combination of feature vectors, so that each classification model has the ability to identify the same disease through the corresponding combination of feature vectors. The test set is used to evaluate the classification performance of each classification model, which is used as the classification performance of each feature vector combination for the same disease, thereby obtaining the classification performance of multimodal medical images formed by different combinations of unimodal medical images for the same disease.
2. The performance evaluation method for multimodal medical images according to claim 1, characterized in that, Classification performance includes accuracy and / or recall.
3. The performance evaluation method for multimodal medical images according to claim 1, characterized in that, The trained feature extractor is used to automatically extract single-modality feature vectors; The training steps for the feature extractor include: Positive sampling is performed on each unimodal medical image sequence in the multimodal medical image sequence in the training set: the unimodal medical image sequence is examined, and images containing lesions are selected as sampling images; A corresponding disease label is set for each lesion on each sampled image, and the sampled images with disease labels are used to build a training dataset. ImageNet1000 is used to pre-train the CNN neural network. The network weights of the CNN neural network are retained, and the last softmax activation layer is replaced with a softmax activation layer that satisfies the number of disease label categories, thereby constructing a feature extractor. The feature extractor is trained using the training dataset.
4. The performance evaluation method for multimodal medical images according to claim 3, characterized in that, The extraction process of a single-modal feature vector includes the following steps: The number of slices in each monomodal medical image sequence is unified to S by sampling or interpolation, and the scanning positions of the corresponding slices are basically consistent. The single-modal medical image sequence is input into the trained feature extractor, and the output of the last fully connected layer is extracted as the feature vector.
5. The performance evaluation method for multimodal medical images according to claim 1, characterized in that, Before feature extraction, the single-modal medical image sequence is preprocessed. The preprocessing steps include image cropping, enhancement, and denoising.
6. A performance evaluation system for multimodal medical images, characterized in that, Includes a multimodal medical image sequence input module, N parallel single-modal feature extraction channels, and a feature combination module. A classification model and a classification performance evaluation module; The multimodal medical image sequence input module is used to input the following data: Multimodal medical image sequences of various single-modal medical image sequences are input into N parallel single-modal feature extraction channels respectively; The single-modal feature extraction channel includes an image preprocessing module and a feature extractor. The feature extractor is obtained by training a CNN neural network using a training set and has the ability to extract single-modal features. The feature combination module is used to combine single-modal feature vectors and generate the resulting... Each feature vector combination is input to... In each classification model; The classification model is used to identify a disease type based on a combination of corresponding feature vectors; The classification performance evaluation module is used to evaluate the classification performance of the classification model based on the classification results of the classification model on the test set, and to evaluate the classification performance of each feature vector combination for the same disease, thereby obtaining the classification performance of multimodal medical images formed by different combinations of single-modal medical images for the same disease.
7. The performance evaluation system for multimodal medical images according to claim 6, characterized in that, The Transformer model or LSTM model is used as the classification model.
8. The performance evaluation system for multimodal medical images according to claim 6, characterized in that, VGG16 / 19, ResNet50 / 101 / 152, InceptionV3, InceptionResnet2, DenseNet169, or DenseNet201 models were used as feature extractors.