Data enhancement method and apparatus based on multimodal language alignment
By using a contrastive learning method to perform image-text aligned encoding on multimodal medical image data, a modality-aligned data augmentation method is generated. This solves the problems of unified representation of multimodal data and missing modalities, improves doctors' diagnostic efficiency and accuracy, and reduces data annotation costs.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- ZHEJIANG LAB
- Filing Date
- 2025-06-04
- Publication Date
- 2026-05-15
AI Technical Summary
Existing multimodal medical imaging data varies greatly between different modalities, and missing modal data is common, making it difficult to effectively unify representation and mine correlations. This limits the efficiency and accuracy of doctors' diagnoses, and the training of existing large models relies on a large amount of manually labeled data.
By collecting multimodal medical data from electronic medical record systems, preprocessing the data, and encoding the images and text, a loss function is constructed using a contrastive learning method to align the modalities and generate image-text pairs, thereby achieving data augmentation for different modalities.
It can generate modality-aligned data without manually labeling text pairs, solving the problem of insufficient data volume in large multimodal models and improving the research efficiency and accuracy of downstream tasks in single-modality models, such as assisted diagnosis and disease prediction.
Smart Images
Figure CN2025098916_15052026_PF_FP_ABST
Abstract
Description
A data augmentation method and apparatus based on multimodal language alignment Technical Field
[0001] This invention relates to the field of multimodal data augmentation technology, and in particular to a data augmentation method and apparatus based on multimodal language alignment. Background Technology
[0002] With the development of artificial intelligence technology, the research and mining of multimodal data has become an increasingly important research field. Multimodal data refers to a data set containing multiple types of information, such as images, text, audio, or video. The applications of multimodal data are wide-ranging, covering multiple fields such as medical diagnosis, autonomous driving, virtual reality, and augmented reality. Multimodal data provides information from different senses or sources, which helps models understand data from multiple perspectives, thus exhibiting stronger generalization ability and robustness when facing new data. However, at the same time, mining multimodal data requires overcoming challenges such as inconsistencies in modal semantics, mismatches in feature dimensions, varying strengths of associations between different modalities, and the potential for missing modal data.
[0003] The goal of multimodal data augmentation is to generate richer and more diverse training samples by processing these different types of data, thereby improving the generalization ability and robustness of the model. With the rise of Large Language Models (LLMs), their powerful language understanding and reasoning capabilities have made them increasingly popular in academia and industry. Advances in LLMs have also inspired researchers to use them as interfaces for multimodal tasks such as visual language learning, audio and speech recognition, and video understanding. Therefore, Multimodal Large Language Models (MLLMs) have attracted considerable attention from researchers. Currently, techniques such as large model fine-tuning and Vision Transformer (ViT) are used for multimodal language alignment training, with the aim of training a general-purpose large multimodal model. This approach requires a large amount of manually labeled image-text pairs as the base data for training. Furthermore, these models and data are designed for general-purpose data, lacking domain knowledge and only capable of simple question answering.
[0004] With the continuous development of technology, medical imaging data mainly includes CT (Computed Tomography) images, ultrasound (US) images, electrocardiogram (ECG), magnetic resonance imaging (MR) images, and nuclear medicine (SPECT, PET) images. Among these, CT images are suitable for imaging hard tissues such as bones and metals, and have high grayscale levels. Ultrasound images are suitable for soft tissues such as the prostate and blood vessels, but have noticeable speckle noise. MRI images have rich grayscale information and are highly effective not only for hard tissues like bones but also for soft tissues such as brain white matter and gray matter; MRI is commonly used for brain examinations. PET images are positron concentration images, highly sensitive for early-stage cancers, but the images themselves are relatively blurry, have low spatial resolution, and cannot clearly reflect the geometric topological information inside the body. An electrocardiogram (ECG) records the changes in electrical activity of the heart during each cardiac cycle from the body surface using an electrocardiograph. These multimodal data provide doctors with invaluable diagnostic information. However, significant differences exist between different images and between images and graphics. Furthermore, in reality, most patients do not have or need to undergo all examinations, making missing modality data a common phenomenon. Unifying the representation of multimodal data, uncovering the relationships between different modalities, and effectively filling in missing modalities are urgent problems to be solved. Therefore, if different modalities can be encoded and missing modalities can be effectively supplemented through multimodal data augmentation methods, it can not only reduce patient treatment costs but also improve the efficiency and accuracy of doctors' diagnoses.
[0005] For multimodal medical image data, this invention proposes a data augmentation method that can directly generate different modality-text pairs and use the similarity between the generated texts for modality alignment or modality data interpolation. This method is based on intuitive generation using language similarity, is highly interpretable, and does not require pre-labeling of text pairs. It can generate different modality-text pairs to address the lack of data volume in current large-scale multimodal model research, while also supporting downstream tasks in different modalities, such as assisted diagnosis, disease prediction, and complication early warning. Summary of the Invention
[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a data augmentation method and apparatus based on multimodal language alignment.
[0007] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a data augmentation method based on multimodal language alignment, comprising the following steps:
[0008] (1) Collect multimodal medical data of patients in the electronic medical record system; wherein, the multimodal medical data includes basic information, diagnostic reports during the observation window and various types of image data;
[0009] (2) Preprocess the multimodal medical data collected in step (1) to reset the size of various images to the same fixed value and clean and standardize the diagnostic reports corresponding to the images;
[0010] (3) Image data encoding and text data encoding are performed on the preprocessed image and its corresponding diagnostic report respectively to obtain the encoded image embedding representation and text embedding representation;
[0011] (4) Modal projection fusion is performed on the encoded image embedding representation and the encoded text embedding representation. Positive and negative samples are learned by using a contrastive learning method to construct the loss function of contrastive learning. The aligned image embedding representation and text embedding representation are determined with the goal of minimizing the loss function of contrastive learning.
[0012] (5) Based on the aligned image embedding representation and text embedding representation determined in step (4), generate a corresponding diagnostic report for the image and a corresponding image for the diagnostic report to obtain aligned image-text pairs.
[0013] Furthermore, in step (1), the basic information includes age and gender; various types of image data include CT images, ultrasound images, electrocardiograms, and MRI images.
[0014] Further, in step (3), the preprocessed image is encoded to obtain the encoded image embedding representation, specifically including:
[0015] (3.1) Slice the preprocessed image to obtain multiple slice images corresponding to the image;
[0016] (3.2) Slice embedding is performed by linear mapping to convolve each slice image into a d-dimensional embedding vector;
[0017] (3.3) For each slice image embedding vector obtained in step (3.2), add a learnable vector z to it. cls Then, the position code and mode code identifier corresponding to the slice image are concatenated into the embedding vector to obtain the concatenated embedding vector;
[0018] (3.4) The concatenated embedding vector is processed through multiple Transformer encoder layers to finally obtain the encoded image embedding representation.
[0019] Furthermore, the Transformer encoder layer includes a multi-head attention mechanism, a feedforward neural network, and a normalization layer, and its input-output process is represented as follows:
[0020] Here, MSA represents multi-head attention mechanism, LN represents normalization layer, MLP represents feedforward neural network, and also represents multilayer perceptron; Z h-1 Z' represents the output of the (h-1)th Transformer encoder layer. h Z represents the intermediate vector of the h-th Transformer encoder layer. h y represents the output of the h-th Transformer encoder layer; h = [1,...,H], where H represents the total number of Transformer encoder layers; y represents the prediction vector output of the last Transformer encoder layer.
[0021] Further, in step (3), the preprocessed diagnostic report is encoded into text data to obtain an encoded text embedding representation, specifically including:
[0022] First, the preprocessed diagnostic report is converted into a binary encoded vector; then, it and its modality identifier are concatenated and input into a bidirectional Transformer encoder. After processing by multiple bidirectional Transformer encoders, the encoded text embedding representation is obtained.
[0023] Furthermore, the bidirectional Transformer encoder includes a bidirectional attention mechanism, a feedforward neural network, and a normalization layer, and its input-output process is represented as: B' h =Bi-A(LN(B w-1 ))+B w-1 B w =MLP(LN(B') w ))+B' w χ=LN(B W )
[0024] Where Bi-A represents bidirectional attention mechanism, B w-1 B' represents the output of the (w-1)th bidirectional Transformer encoder. w B represents the intermediate vector of the w-th bidirectional Transformer encoder. w χ represents the output of the w-th bidirectional Transformer encoder, where w = [1,...,W], W represents the total number of W bidirectional Transformer encoders, and χ represents the encoded text embedding representation output by the last bidirectional Transformer encoder.
[0025] Further, in step (4), the expression for the loss function of the contrastive learning is:
[0026] Among them, L I,T Y represents the loss function for contrastive learning, where N is the total number of samples, and Y is the loss function for contrastive learning. i χ represents the image embedding representation of the i-th sample. j Let S(Y) represent the text embedding representation of the j-th sample. i ,χ i ) represents Y i and χ i Cosine similarity between them, S(Y) i ,χ j ) represents Y i and χ j The cosine similarity between them, τ is a hyperparameter used to adjust the probability distribution.
[0027] Further, in step (5), generating a corresponding diagnostic report for the image specifically includes: first, encoding the given image data to obtain the encoded image embedding representation; then calculating the cosine similarity between the image embedding representation and all text embedding representations determined in step (4), and selecting the text embedding representation corresponding to the maximum cosine similarity as the diagnostic report text corresponding to the image;
[0028] The specific steps for generating the corresponding image for the diagnostic report include: first, encoding the given diagnostic report into text data to obtain the encoded text embedding representation; then, calculating the cosine similarity between the text embedding representation and all image embedding representations determined in step (4), and selecting the image embedding representation with the highest cosine similarity as the image corresponding to the diagnostic report.
[0029] Furthermore, following step (5), the following steps are also included:
[0030] Different post-processing procedures are designed according to different downstream tasks. Based on the aligned image-text pairs obtained in step (5), the corresponding downstream tasks are executed.
[0031] A second aspect of this invention provides a data augmentation apparatus based on multimodal language alignment, used to implement the above-described data augmentation method based on multimodal language alignment, comprising:
[0032] The data acquisition module is used to acquire multimodal medical data of patients in the electronic medical record system; wherein, the multimodal medical data includes basic information, diagnostic reports during the observation window, and various types of image data;
[0033] The data preprocessing module is used to preprocess the multimodal medical data acquired by the data acquisition module, in order to reset the size of various images to the same fixed value and to clean and standardize the diagnostic reports corresponding to the images.
[0034] Each modal data encoding module includes an image data encoding module and a text data encoding module. The image data encoding module is used to encode the preprocessed image to obtain an encoded image embedding representation. The text data encoding module is used to encode the preprocessed diagnostic report to obtain an encoded text embedding representation.
[0035] The projection module performs modal projection fusion of the encoded image embedding representation and the encoded text embedding representation. It learns positive and negative samples using a contrastive learning method to construct a contrastive learning loss function; and determines the aligned image embedding representation and text embedding representation with the objective of minimizing the contrastive learning loss function.
[0036] The decoding module is used to generate a corresponding diagnostic report for the image and a corresponding image for the diagnostic report based on the aligned image embedding representation and text embedding representation determined by the projection module, so as to obtain aligned image-text pairs.
[0037] The beneficial effects of this invention are as follows: This invention uses a multimodal encoding method to encode different image modalities and text, which not only distinguishes the modalities between images and text but also differentiates images of different modalities. Furthermore, this invention aligns multimodal medical image data and text data at the language level, and proposes a decoding method for generating image-text pairs based on language similarity. Based on the generated image-text pairs, multimodal large-scale model research can be conducted. For research tasks with limited single-modal data, images of a specified modality can be generated, solving the problem of insufficient data volume in multimodal large-scale model research and supporting downstream single-modal tasks such as assisted diagnosis, disease prediction, and complication early warning. Attached Figure Description
[0038] Figure 1 is a flowchart of the data augmentation method based on multimodal language alignment according to the present invention;
[0039] Figure 2 is a flowchart of the framework of each modal data encoding module in this invention;
[0040] Figure 3 is a flowchart of the projection module in this invention;
[0041] Figure 4 is a flowchart of the decoding module in this invention;
[0042] Figure 5 is a schematic diagram of the data augmentation device based on multimodal language alignment according to the present invention. Detailed Implementation
[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.
[0044] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0045] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."
[0046] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.
[0047] Referring to Figure 1, the data augmentation method based on multimodal language alignment of the present invention specifically includes the following steps:
[0048] (1) Collect multimodal medical data of patients in the electronic medical record system; among which, multimodal medical data includes basic information, diagnostic reports during the observation window and various image data.
[0049] Furthermore, basic information includes age and gender, and various types of image data include CT images, ultrasound images, electrocardiograms, and MRI images.
[0050] (2) The multimodal medical data collected in step (1) is preprocessed to reset the size of various images to the same fixed value and to clean and standardize the diagnostic reports corresponding to the images.
[0051] Specifically, the size of all types of images acquired in step (1) is reset to the same fixed value. The diagnostic reports corresponding to the images are cleaned and standardized to unify the fixed meaning. For example, different doctors may abbreviate cancer as CA, Ca, ca, etc., and these are unified as CA during data preprocessing. The patient set is represented by V, the patient's image data by I, the patient's diagnostic report data by T, and n represents the maximum index of the corresponding category of data possessed by the patient. The value of n will change with different categories of medical data. Therefore, the patient can be formally represented as V = {I; T}, where I = {I... C ,I U ,I E ,I M ,I P} represents the patient's image data, I C I U I E I M and I P These represent CT images, ultrasound images, electrocardiogram images, MRI images, and nuclear medicine images, respectively, where T = {T C ,T U ,T E ,T M ,T P} represents the diagnostic report data corresponding to the patient, T C T U T E T M and T P These represent the diagnostic report data corresponding to CT images, ultrasound images, electrocardiogram images, MRI images, and nuclear medicine images, respectively. The i-th patient can be represented as... For medical data of different modalities, data gaps are allowed, i.e., patient v i The value I Ci It may be empty, so replace it with 0. The same applies to other modalities.
[0052] (3) Image data encoding and text data encoding are performed on the preprocessed image and its corresponding diagnostic report respectively to obtain the encoded image embedding representation and text embedding representation, as shown in Figure 2.
[0053] Furthermore, the preprocessed image is encoded to obtain the encoded image embedding representation, as shown in Figure 2, which specifically includes:
[0054] (3.1) Slice the preprocessed image to obtain multiple slice images corresponding to the image.
[0055] Specifically, the size of the preprocessed image is set to H*W, and it is sliced to obtain multiple slice images. The size of each slice image is N*N. After slicing, P = (H / N)*(W / N) slice images can be obtained. The size can be adjusted according to actual needs. In this embodiment, N is set to 16, that is, the original image 224*224*3 is cut into a sequence of 14*14 slice images of size 16*16*3.
[0056] (3.2) Patch embedding is performed by linear mapping to convolve each slice image into a d-dimensional embedding vector.
[0057] Specifically, slice embedding is performed through linear mapping to convolve each 3D 16*16 slice image into a d-dimensional embedding vector. That is, the original 16*16*3 slice image is convolved into an embedding vector of length 768, and the whole image becomes 14*14*768 in dimension. When flattened, it becomes a 196*768 two-dimensional matrix. Therefore, the whole image can be represented by a two-dimensional matrix.
[0058] (3.3) For each slice image embedding vector obtained in step (3.2), add a learnable vector z to it. cls Then, the position code and mode code identifier corresponding to the slice image are concatenated into the embedding vector to obtain the concatenated embedding vector.
[0059] Specifically, to represent the global information of the matrix, the BERT (Bidirectional Encoder Representations from Transformers) model adds a learnable vector [CLS]z to the embedding vector. cls Next, the positional encoding (PE) and modal indicator (MI) of the slice image are concatenated into the embedding vector, resulting in the concatenated embedding vector Z, as shown in formula (1). The concatenated embedding vector Z contains the information of the original slice image after slice embedding and the learnable vector z. cls Location code (PE) and mode code identifier (MI).
[0060] Where Z represents the concatenated embedding vector, I represents the patient's image data, patch is the slicing function, linear is the linear mapping function, and z clsLet [CLS] be a learnable vector, PE be the positional encoding, MI be the modality encoding identifier, d be the total dimension of the positional encoding of the word vector (i.e., the embedding vector corresponding to the current slice image), p be the p-th word in the sentence (i.e., the p-th slice image in the whole image), i = [1, d / 2] be the dimension index of the positional encoding vector, and E be the positional encoding vector. m Let E represent the learnable mode vector, based on which the learnable mode vector E is... m Modal coding is performed, with the subscript m representing the m-th mode.
[0061] (3.4) The concatenated embedding vector is processed through multiple Transformer encoder layers to obtain the encoded image embedding representation; each Transformer encoder layer includes components such as a multi-head attention mechanism, a feedforward neural network, and a normalization layer, and its input and output process is represented by formula (2):
[0062] Here, MSA represents multi-head attention mechanism, LN represents normalization layer, MLP represents feedforward neural network, and also represents multilayer perceptron; Z h-1 Z' represents the output of the (h-1)th Transformer encoder layer. h Z represents the intermediate vector of the h-th Transformer encoder layer. h Let y represent the output of the h-th Transformer encoder layer; h = [1,...,H], where H represents the total number of Transformer encoder layers; y represents the prediction vector output by the last Transformer encoder layer. Combining the prediction vectors corresponding to all slice images yields the image embedding representation Y for the entire image.
[0063] It should be understood that, In the image, the part before position 0 is a learnable category vector. After encoding, the information at this position represents the overall representation of the image, while the other positions are the encoded representations of the corresponding images.
[0064] Further, the preprocessed diagnostic report is encoded into text data to obtain the encoded text embedding representation, as shown in Figure 2. Specifically, this includes: first, converting the preprocessed diagnostic report into a binary encoded vector; then concatenating it with its modality identifier and inputting it into a bidirectional Transformer encoder. After processing by multiple bidirectional Transformer encoders, the encoded text embedding representation is obtained. Each bidirectional Transformer encoder includes components such as a bidirectional attention mechanism, a feedforward neural network, and a normalization layer. Its input and output process is shown in formula (3):
[0065] Where Bi-A represents bidirectional attention mechanism, B w-1 B' represents the output of the (w-1)th bidirectional Transformer encoder. w B represents the intermediate vector of the w-th bidirectional Transformer encoder. w χ represents the output of the w-th bidirectional Transformer encoder, where w = [1,...,W], W represents the total number of W bidirectional Transformer encoders, and χ represents the encoded text embedding representation output by the last bidirectional Transformer encoder.
[0066] It should be understood that by using a bidirectional Transformer encoder, contextual information in text data can be captured to encode text-based diagnostic reports. First, the text is converted into a binary encoded vector. Then, this vector, along with its modality identifier, is input into the encoder. After passing through a bidirectional attention mechanism, a feedforward neural network, and a normalization operation, the encoded vector of the text is obtained, i.e., the encoded text embedding representation χ.
[0067] (4) Modal projection fusion is performed on the encoded image embedding representation and the encoded text embedding representation. Positive and negative samples are learned by using contrastive learning method to construct the loss function of contrastive learning. The aligned image embedding representation and text embedding representation are determined with the goal of minimizing the loss function of contrastive learning.
[0068] Specifically, the encoded image embedding representation Y and the encoded text embedding representation χ are fused using modal projection, as shown in Figure 3. The blue diagonal lines represent positive samples where the image and text are identical, while the others are negative samples. A contrastive learning method is used to learn the positive and negative samples of both modalities, maximizing the similarity between each image and its corresponding text description while minimizing its similarity to other text descriptions. To this end, the contrastive learning method defines the loss function as L. I,T As shown in Equation (4), the loss function of this contrastive learning includes two terms: the first term is the image-to-text contrastive loss (i.e., the first log), and the second term is the text-to-image contrastive loss (i.e., the second log). The average of the contrastive losses in these two directions is the loss function of contrastive learning. By minimizing the loss function of contrastive learning, the aligned image embedding representation and text embedding representation can be determined for subsequent inference tasks.
[0069] Among them, L I,T Y represents the loss function for contrastive learning, where N is the total number of samples, and Y is the loss function for contrastive learning. i Let χ represent the image embedding representation of the i-th sample. jLet S(Y) represent the text embedding representation of the j-th sample. i ,χ i ) represents Y i and χ i The cosine similarity between positive sample pairs (i.e., the cosine similarity between positive sample pairs), S(Y) i ,χ j ) represents Y i and χ j The cosine similarity between positive and negative samples is τ, which is a hyperparameter used to adjust the probability distribution.
[0070] (5) Based on the aligned image embedding representation and text embedding representation determined in step (4), generate a corresponding diagnostic report for the image and a corresponding image for the diagnostic report to obtain aligned image-text pairs, which helps to reduce the cost of manually labeling image-text pairs.
[0071] Further, as shown in Figure 4, generating a corresponding diagnostic report for the image specifically includes: first, encoding the given image data to obtain the encoded image embedding representation; then calculating the cosine similarity between the image embedding representation and all text embedding representations determined in step (4), and selecting the text embedding representation corresponding to the maximum cosine similarity as the diagnostic report text corresponding to the image.
[0072] Further, generating a corresponding image for the diagnostic report specifically includes: first, encoding the given diagnostic report into text data to obtain the encoded text embedding representation; then calculating the cosine similarity between the text embedding representation and all image embedding representations determined in step (4), and selecting the image embedding representation corresponding to the maximum cosine similarity as the image corresponding to the diagnostic report.
[0073] Furthermore, after step (5), it also includes: designing different post-processing procedures according to different downstream tasks, which is the reasoning application of the whole method. Based on the aligned image-text pairs obtained in step (5), the corresponding downstream tasks are executed, such as assisted diagnosis, disease prediction, complication warning and other downstream tasks.
[0074] It is worth mentioning that the present invention also provides a data augmentation device based on multimodal language alignment, as shown in FIG5. The device includes a data acquisition module, a data preprocessing module, a modal data encoding module, a projection module, and a decoding module; wherein, the modal data encoding module includes an image data encoding module and a text data encoding module.
[0075] In this embodiment, the data acquisition module is used to collect multimodal medical data of patients in the electronic medical record system; wherein, the multimodal medical data includes basic information, diagnostic reports during the observation window, and various types of image data, etc.
[0076] In this embodiment, the data preprocessing module is used to preprocess the multimodal medical data collected by the data acquisition module to reset the size of various images to the same fixed value and to clean and standardize the diagnostic reports corresponding to the images.
[0077] In this embodiment, the image data encoding module is used to encode the preprocessed image to obtain an encoded image embedding representation. The text data encoding module is used to encode the preprocessed diagnostic report to obtain an encoded text embedding representation.
[0078] In this embodiment, the projection module is used to perform modal projection fusion of the encoded image embedding representation and the encoded text embedding representation, and to learn positive and negative samples by using a contrastive learning method to construct a contrastive learning loss function; with the goal of minimizing the contrastive learning loss function, the aligned image embedding representation and text embedding representation are determined.
[0079] In this embodiment, the decoding module is used to generate a corresponding diagnostic report for the image and a corresponding image for the diagnostic report based on the aligned image embedding representation and text embedding representation determined by the projection module, so as to obtain aligned image-text pairs.
[0080] The data augmentation method and apparatus based on multimodal language alignment of the present invention will be described in detail below with reference to embodiments, and the purpose and effects of the present invention will become more apparent.
[0081] The data augmentation method specifically includes:
[0082] S1. The data acquisition module collects basic information, diagnostic reports during the observation window, and various image data of 10,000 patients from a medical institution.
[0083] S2. The data preprocessing module preprocesses the data collected by the data acquisition module, resizing the image size of each modality to 224*224*3, cleaning and standardizing the corresponding diagnostic reports, and unifying the representation of meaning (for example, different doctors may abbreviate cancer as CA, Ca, ca, etc., which are unified as CA during data preprocessing). The patient set is represented by V, the patient's image information by I, the patient's diagnostic information by T, and N represents the maximum index of the corresponding category of data possessed by the patient. The value of N varies with different categories of data. A patient can be formally represented as V = {I; T}, where I = {I... C ,I U ,I E ,I M ,I P} represent CT images I CUltrasound image I U Electrocardiogram image I E Nuclear magnetic resonance image I M and nuclear medicine images I P T = {T} C ,T U ,T E ,T M ,T P} represents the corresponding diagnostic information. For patient i, it can be represented as: For data from different modalities, missing data is allowed, i.e., the value I in patient i is allowed. Ci It may be empty, and the same applies to other modalities.
[0084] The image and text data encoding modules of the S3 multimodal data encoding module encode the preprocessed medical images and diagnostic reports respectively. Modality identification is performed and encoded based on the different modalities of the medical image data. First, the image is sliced, with a slice size of 16, meaning the original 224*224*3 image is cut into a sequence of 14*14 slices, each consisting of a 16*16*3 slice. Next, each 3D slice is convolved into a d-dimensional vector using a linear mapping of Patch Embedding. The original 16*16*3 slice becomes a vector Z of length 768, and the entire image becomes a 14*14*768 dimension. Flattening this results in a 196*768 two-dimensional matrix. Furthermore, a learnable vector [CLS] is concatenated to the vector header. Then, the image's positional encoding and modality encoding identifier are concatenated into vector Z and input into the Transformer encoder layer for encoding. Meanwhile, the standardized diagnostic report is converted into a binary identifier, then modal identifiers are added, and it is input into the BERT model for text encoding to obtain the encoding vector χ.
[0085] S4. The projection module performs modal projection fusion on the image embedding representation Z and the text embedding representation χ obtained from the multimodal data encoding module. Contrastive learning is used to learn the positive and negative pairs of the two modalities, and the loss function for contrastive learning is L. I,T By minimizing the loss function L I,T This maximizes the similarity between each image and its corresponding text description, while minimizing the similarity between each image and other text descriptions.
[0086] S5. In the decoding module, new image-text pairs are generated. Images with missing diagnostic reports are encoded, and then the similarity between the image and text embeddings is calculated. The most similar ultrasound report is output. For image data with a small amount of single-modal data, given an ultrasound diagnostic image, the text is encoded, and then the most similar image is calculated and output.
[0087] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.
Claims
1. A data augmentation method based on multimodal language alignment, characterized in that, Includes the following steps: (1) Collect multimodal medical data of patients in the electronic medical record system; wherein, the multimodal medical data includes basic information, diagnostic reports during the observation window and various types of image data; (2) Preprocess the multimodal medical data collected in step (1) to reset the size of various images to the same fixed value and clean and standardize the diagnostic reports corresponding to the images; (3) Image data encoding and text data encoding are performed on the preprocessed image and its corresponding diagnostic report respectively to obtain the encoded image embedding representation and text embedding representation; (4) Modal projection fusion is performed on the encoded image embedding representation and the encoded text embedding representation. Positive and negative samples are learned by using a contrastive learning method to construct the loss function of contrastive learning. The aligned image embedding representation and text embedding representation are determined with the goal of minimizing the loss function of contrastive learning. (5) Based on the aligned image embedding representation and text embedding representation determined in step (4), generate a corresponding diagnostic report for the image and a corresponding image for the diagnostic report to obtain aligned image-text pairs.
2. The data augmentation method based on multimodal language alignment according to claim 1, characterized in that, In step (1), the basic information includes age and gender; various types of image data include CT images, ultrasound images, electrocardiograms and MRI images.
3. The data augmentation method based on multimodal language alignment according to claim 1, characterized in that, In step (3), the preprocessed image is encoded to obtain the encoded image embedding representation, specifically including: (3.1) Slice the preprocessed image to obtain multiple slice images corresponding to the image; (3.2) Slice embedding is performed by linear mapping to convolve each slice image into a d-dimensional embedding vector; (3.3) For each slice image embedding vector obtained in step (3.2), add a learnable vector z to it. cls Then, the position code and mode code identifier corresponding to the slice image are concatenated into the embedding vector to obtain the concatenated embedding vector; (3.4) The concatenated embedding vector is processed through multiple Transformer encoder layers to finally obtain the encoded image embedding representation.
4. The data augmentation method based on multimodal language alignment according to claim 3, characterized in that, The Transformer encoder layer includes a multi-head attention mechanism, a feedforward neural network, and a normalization layer. Its input-output process is represented as: Z' h =MSA(LN(Z) h-1 ))+Z h-1 Z h =MLP(LN(Z') h ))+Z h ' Here, MSA represents multi-head attention mechanism, LN represents normalization layer, MLP represents feedforward neural network, and also represents multilayer perceptron; Z h-1 Z' represents the output of the (h-1)th Transformer encoder layer. h Z represents the intermediate vector of the h-th Transformer encoder layer. h y represents the output of the h-th Transformer encoder layer; h = [1,...,H], where H represents the total number of Transformer encoder layers; y represents the prediction vector output of the last Transformer encoder layer.
5. The data augmentation method based on multimodal language alignment according to claim 1, characterized in that, In step (3), the preprocessed diagnostic report is encoded into text data to obtain an encoded text embedding representation, specifically including: First, the preprocessed diagnostic report is converted into a binary encoded vector; then, it and its modality identifier are concatenated and input into a bidirectional Transformer encoder. After processing by multiple bidirectional Transformer encoders, the encoded text embedding representation is obtained.
6. The data augmentation method based on multimodal language alignment according to claim 5, characterized in that, The bidirectional Transformer encoder includes a bidirectional attention mechanism, a feedforward neural network, and a normalization layer. Its input-output process is represented as: B' h =Bi-A(LN(B w-1 ))+B w-1 B w =MLP(LN(B') w ))+B' w χ=LN(B W ) Where Bi-A represents bidirectional attention mechanism, B w-1 B' represents the output of the (w-1)th bidirectional Transformer encoder. w B represents the intermediate vector of the w-th bidirectional Transformer encoder. w χ represents the output of the w-th bidirectional Transformer encoder, where w = [1,...,W], W represents the total number of W bidirectional Transformer encoders, and χ represents the encoded text embedding representation output by the last bidirectional Transformer encoder.
7. The data augmentation method based on multimodal language alignment according to claim 1, characterized in that, In step (4), the expression for the loss function of the contrastive learning is: Among them, L I,T Y represents the loss function for contrastive learning, where N is the total number of samples, and Y is the loss function for contrastive learning. i χ represents the image embedding representation of the i-th sample. j Let S(Y) represent the text embedding representation of the j-th sample. i ,χ i ) represents Y i and χ i Cosine similarity between them, S(Y) i ,χ j ) represents Y i and χ j The cosine similarity between them, τ is a hyperparameter used to adjust the probability distribution.
8. The data augmentation method based on multimodal language alignment according to claim 1, characterized in that, In step (5), generating a corresponding diagnostic report for the image specifically includes: first, encoding the given image data to obtain the encoded image embedding representation; then calculating the cosine similarity between the image embedding representation and all text embedding representations determined in step (4), and selecting the text embedding representation corresponding to the maximum cosine similarity as the diagnostic report text corresponding to the image; The specific steps for generating the corresponding image for the diagnostic report include: first, encoding the given diagnostic report into text data to obtain the encoded text embedding representation; then, calculating the cosine similarity between the text embedding representation and all image embedding representations determined in step (4), and selecting the image embedding representation with the highest cosine similarity as the image corresponding to the diagnostic report.
9. The data augmentation method based on multimodal language alignment according to claim 1, characterized in that, Following step (5), the following is also included: Different post-processing procedures are designed according to different downstream tasks. Based on the aligned image-text pairs obtained in step (5), the corresponding downstream tasks are executed.
10. A data augmentation apparatus based on multimodal language alignment, used to implement the data augmentation method based on multimodal language alignment as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire multimodal medical data of patients in the electronic medical record system; wherein, the multimodal medical data includes basic information, diagnostic reports during the observation window, and various types of image data; The data preprocessing module is used to preprocess the multimodal medical data acquired by the data acquisition module, in order to reset the size of various images to the same fixed value and to clean and standardize the diagnostic reports corresponding to the images. Each modal data encoding module includes an image data encoding module and a text data encoding module. The image data encoding module is used to encode the preprocessed image to obtain an encoded image embedding representation. The text data encoding module is used to encode the preprocessed diagnostic report to obtain an encoded text embedding representation. The projection module performs modal projection fusion of the encoded image embedding representation and the encoded text embedding representation. It learns positive and negative samples using a contrastive learning method to construct a contrastive learning loss function; and determines the aligned image embedding representation and text embedding representation with the objective of minimizing the contrastive learning loss function. The decoding module is used to generate a corresponding diagnostic report for the image and a corresponding image for the diagnostic report based on the aligned image embedding representation and text embedding representation determined by the projection module, so as to obtain aligned image-text pairs.