Medical image generation method and apparatus based on dual-modal fusion, and device and medium

Abstract

A medical image generation method and apparatus based on dual-modal fusion, and a device and a medium, which are applied to the digital medical treatment field. An image encoder and a text encoder are used to extract a visual feature of an image and a semantic feature of text, and then by means of contrastive learning, the visual feature of the image and the semantic feature of the text are mapped to a shared semantic space for cross-modal alignment, and fusion is performed to obtain a cross-modal feature; a dual-branch encoder is used to perform multi-scale feature extraction, wherein one branch involves using convolution to capture hierarchical semantics, and the other branch involves using maximum pooling to retain local saliency, and fusion is performed layer by layer to obtain a fused encoded feature; the cross-modal feature is projected as a spatial feature map by means of a multi-layer perceptron, the spatial feature map is concatenated with the fused encoded feature, and then a decoder performs up-sampling and reconstruction; and a contrastive loss and a structural similarity loss are jointly optimized to implement end-to-end training, thereby improving the SSIM index of a generated medical image.

Description

Methods, devices, equipment, and media for generating medical images based on dual-modal fusion

[0001] This application is based on and claims priority to Chinese Patent Application No. CN202510208510.6, filed on February 24, 2025, entitled "Method, Apparatus, Device and Medium for Medical Image Generation Based on Dual-Modal Fusion", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of medical image generation, and in particular to a method, apparatus, device and medium for generating medical images based on dual-modal fusion. Background Technology

[0003] Currently, generative models such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) are widely used in medical image generation. However, the inventors have found that these traditional techniques, or even just one technique, have limited performance when dealing with long-tailed medical data. The scarcity of rare disease case data greatly limits the training and generation capabilities of deep learning models, and traditional techniques, or even just one technique, are particularly inadequate when combining image and text information, lacking the correlation between text and image, which reduces the quality and consistency of rare case images. Therefore, there is an urgent need for a method to improve the SSIM (Simplified Simulation Metric) of medical images generated by medical image generation models. Summary of the Invention

[0004] The embodiments of this application provide a medical image generation method, apparatus, device and medium based on dual-modal fusion, which aims to solve the problem of low quality of medical images generated by existing medical image generation models.

[0005] In a first aspect, embodiments of this application provide a medical image generation method based on dual-modal fusion, comprising:

[0006] Acquire medical images and corresponding text descriptions, and perform standardized preprocessing to obtain preprocessed images and text sequences;

[0007] Image features are extracted from the preprocessed image using an image encoder, and text features are extracted from the text sequence using a text encoder. The image features and text features are then mapped to a shared semantic space and fused using contrastive learning to obtain cross-modal features.

[0008] Encoded features are obtained by multi-scale feature extraction of the medical image using a dual-channel encoder;

[0009] The cross-modal features are fused with the encoded features to obtain the decoder input features, and the target image is generated by the decoder based on the decoder input features;

[0010] The target image is used to construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the dual encoder, and the decoder.

[0011] Secondly, embodiments of this application also provide a medical image generation device based on dual-modal fusion, comprising: an information preprocessing unit, used to acquire a medical image and its corresponding text description, and perform standardized preprocessing to obtain a preprocessed image and a text sequence; a cross-modal coding fusion unit, used to extract image features from the preprocessed image through an image encoder, extract text features from the text sequence through a text encoder, and map the image features and the text features to a shared semantic space through contrastive learning and fuse them to obtain cross-modal features; a dual-channel encoder working unit, used to extract multi-scale features from the medical image through a dual-channel encoder to obtain fused coding features; an image decoding unit, used to generate decoder input features by fusing the cross-modal features and the coding features, and generate a target image through a decoder based on the decoder input features; and an optimization training unit, used to construct a joint loss function using the target image to jointly optimize the network parameters of the image encoder, the text encoder, the dual-channel encoder, and the decoder.

[0012] Thirdly, embodiments of this application also provide a computer device, including a memory and a processor connected to the memory; the memory is used to store a computer program; wherein the processor is used to run the computer program stored in the memory to perform the following steps: acquiring a medical image and a corresponding text description, performing standardized preprocessing to obtain a preprocessed image and a text sequence; extracting image features from the preprocessed image using an image encoder, extracting text features from the text sequence using a text encoder, mapping the image features and the text features to a shared semantic space through contrastive learning and fusing them to obtain cross-modal features; extracting multi-scale features from the medical image using a dual-channel encoder to obtain fused encoded features; fusing the cross-modal features and the encoded features to obtain decoder input features, generating a target image through a decoder based on the decoder input features; and constructing a joint loss function using the target image to jointly optimize the network parameters of the image encoder, the text encoder, the dual-channel encoder, and the decoder.

[0013] Fourthly, embodiments of this application also provide a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the following steps: acquiring a medical image and its corresponding text description, performing standardized preprocessing to obtain a preprocessed image and a text sequence; extracting image features from the preprocessed image using an image encoder, extracting text features from the text sequence using a text encoder, mapping the image features and the text features to a shared semantic space through contrastive learning and fusing them to obtain cross-modal features; extracting multi-scale features from the medical image using a dual-channel encoder to obtain fused encoded features; fusing the cross-modal features and the encoded features to obtain decoder input features, generating a target image through a decoder based on the decoder input features; and constructing a joint loss function using the target image to jointly optimize the network parameters of the image encoder, the text encoder, the dual-channel encoder, and the decoder.

[0014] Compared with the prior art, the beneficial effects of this application are:

[0015] In the technical solution of this application, visual features of medical images and semantic features of text descriptions are extracted by an image encoder and a text encoder, respectively. Contrastive learning is used to map the two types of features to a shared semantic space to achieve cross-modal alignment. A dual-path encoder is constructed to perform multi-scale feature extraction on the medical images: the first path captures hierarchical semantic information through a convolutional neural network, and the second path retains local salient features through max pooling. The outputs of the two paths are fused layer by layer to construct spatial context features. The cross-modal features are projected into a spatial feature map by a multilayer perceptron, concatenated with the encoded features, and then input into the decoder for upsampling reconstruction. The contrast loss and structural similarity loss are jointly optimized to achieve end-to-end training. Ultimately, the SSIM index of the generated medical images is improved. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is a flowchart of the medical image generation method based on dual-modal fusion provided in this application;

[0018] Figure 2 is a first sub-flowchart of the medical image generation method based on dual-modal fusion provided in this application;

[0019] Figure 3 is a second sub-flowchart of the medical image generation method based on dual-modal fusion provided in this application;

[0020] Figure 4 is the third sub-flowchart of the medical image generation method based on dual-modal fusion provided in this application;

[0021] Figure 5 is the fourth sub-flowchart of the medical image generation method based on dual-modal fusion provided in this application;

[0022] Figure 6 is the fifth sub-flowchart of the medical image generation method based on dual-modal fusion provided in this application;

[0023] Figure 7 is a schematic block diagram of the unit of the medical image generation device based on dual-modal fusion provided in this application;

[0024] Figure 8 is a schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terminology used in this specification is for the purpose of describing medical embodiments only and is not intended to limit the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] This application addresses the problem of low-quality medical images generated by existing medical image generation models by providing a medical image generation method, apparatus, device, and medium based on bimodal fusion. Referring to Figures 1 to 6, the medical image generation method based on bimodal fusion includes the following steps.

[0030] S110. Obtain medical images and corresponding text descriptions, and perform standardized preprocessing to obtain preprocessed images and text sequences;

[0031] When a medical image generation model generates medical images to meet specific needs, it first needs to obtain the source files for generating the desired images. In the case of this method, the source files refer to the medical image data that can be processed by the model and the text descriptions used by medical personnel to refine the image descriptions or specify image generation requirements. Specifically, medical images are typically two-dimensional or three-dimensional images obtained from medical imaging techniques such as CT scans, MRI scans, or X-rays. The text descriptions can be key symptom descriptions from image aid reports, medical records, or text conclusions provided by models for medical record analysis. The medical images are then processed to enable image extraction via an encoder.

[0032] Further, referring to Figure 2, step S110 includes:

[0033] S111. Obtain the medical image and the corresponding text description through the interactive platform;

[0034] S112. Perform format standardization, numerical standardization, and size unification on the medical image to obtain the preprocessed image;

[0035] S113. Extract key information from the text description, perform word segmentation and length normalization to obtain the text sequence;

[0036] The interactive platform refers to software used by medical personnel to input and output data into the model. Through this software, medical personnel can input medical images and text descriptions into the model. Specifically, medical images are stored in DICOM format, and text descriptions exist in the form of electronic reports. For ease of use, the model also includes a pre-installed program for extracting key symptom descriptions from radiology reports, reducing the workload for staff.

[0037] In the preprocessing of medical images, format standardization can be performed as needed, such as converting original DICOM files to PNG format to eliminate manufacturer-specific encoding differences and retain grayscale information. Numerical standardization can also be performed, such as normalization and Z-score standardization. Image size can also be standardized, for example, by adjusting the image resolution to 224×224 using bicubic interpolation to match model input requirements. Finally, a preprocessed image can be generated.

[0038] During the preprocessing of text descriptions, information extraction can be performed as needed, such as locating symptom description fragments (e.g., "periosteal hyperplasia") from radiology reports and filtering irrelevant text. Tokenization can be performed on the text descriptions as needed, such as using the BERT-base tokenizer to generate WordPiece token sequences and adding [CLS] start characters and [SEP] delimiters. Length regularization can be performed on the text descriptions as needed, such as padding or truncating the text sequence to a fixed length of 64 tokens, padding with zeros if necessary. Finally, a text sequence is generated.

[0039] S120. Image features are extracted from the preprocessed image using an image encoder, and text features are extracted from the text sequence using a text encoder. The image features and text features are mapped to a shared semantic space and fused using contrastive learning to obtain cross-modal features.

[0040] After acquiring the preprocessed image and text sequences, image and text features are obtained through image encoders and text encoders. Specifically, these image and text features refer to image and text embeddings. Two mapping networks are trained through contrastive learning to map image and text embeddings of different dimensions to a shared semantic space. Specifically, the image and text mapping networks perform linear transformations and normalization on the embeddings, mapping them to vectors in the shared space. After obtaining image and text vectors of the same dimension, they are directly added together to fuse the bimodal information and obtain the final input, i.e., cross-modal features.

[0041] Further, referring to Figure 3, step S120 includes:

[0042] S121. Input the preprocessed image into the image encoder to perform multi-level feature extraction to obtain the image embedding vector;

[0043] S122. Input the text sequence into the text encoder for contextual semantic encoding to obtain the text embedding vector.

[0044] S123. The image embedding vector and the text embedding vector are mapped to a shared semantic space through a mapping network trained by contrastive learning;

[0045] S124. Normalize the image embedding vector and the text embedding vector to obtain normalized image features and normalized text features;

[0046] S125. The normalized image features and the normalized text features are added and fused element by element to obtain cross-modal features.

[0047] The preprocessed image is input into the image encoder. For example, the preprocessed image is a normalized grayscale CT image (224×224×1) of a diseased bone joint, and the image encoder is ResNet-50 or EfficientNet-B5. Through hierarchical convolution, after passing through feature layers and semantic layers, the final output flattens the highest-level features into a 2048-dimensional vector as the image embedding. Dynamic layer partitioning can be used in the convolutional layers to achieve multi-resolution image data processing. Furthermore, deformable convolution can be used, dynamically adjusting the sampling position of the convolution kernel to enhance adaptability to lesion deformation.

[0048] The text sequence is input into a text encoder, which performs contextual semantic encoding. For example, the input is a padded 64-token sequence, and the BERT model or BioClinicalBERT is selected as the text editor based on the difficulty of understanding the medical terminology. Contextual semantic encoding mechanisms may include self-attention computation or positional encoding injection. Finally, a text embedding is generated, for example, extracting the 768-dimensional vector of the [CLS] token as the text embedding.

[0049] By comparing and contrasting trained image and text mapping networks, image and text information of different dimensions are mapped to the same shared space, the dimension of which is preset. Image embeddings and text embeddings are subjected to linear transformations and L2 normalization or batch normalization scaling to obtain normalized image and text features, both of which are image and text vectors of the same dimension. Then, the bimodal information is fused element-wise to obtain a single vector, which is the cross-modal feature. Specifically, deep fusion can also be used, employing a multi-layer fusion strategy to fuse image and text features layer by layer at different levels, rather than directly at the final layer. This allows information to be fused at different levels of abstraction, improving model performance.

[0050] S130. Multi-scale feature extraction is performed on the medical image using a dual-channel encoder to obtain encoded features;

[0051] The medical image is used as the input dual-channel encoder. Features are extracted through the first and second coding paths respectively. At each coding level, the image semantic features and the level features are added and fused element by element to finally extract the coded features.

[0052] Further, referring to Figure 4, step S130 includes:

[0053] S131. Use the medical image as the input of the dual-channel encoder;

[0054] S132. The image semantic features are extracted layer by layer using a convolutional neural network through the first encoding path of the dual encoder;

[0055] S133. Hierarchical features are obtained by downsampling through the second encoding path of the dual encoder using the max pooling layer.

[0056] S134. At each coding level, the image semantic features of the current level are added element-wise to the layer features to obtain the fused coding features.

[0057] The same image copy is input into two independent pathways for processing. One encoding pathway extracts semantic features of the image layer by layer through a convolutional neural network (CNN). The input medical image is processed through multiple convolutional layers, and each layer extracts features at different levels. These features can reflect important structural information in the image.

[0058] Another encoding pathway obtains hierarchical features through step-by-step downsampling using max pooling layers. Max pooling reduces the image size by sliding a small pooling window across the image's feature map and selecting the maximum value within each window. After pooling, the image's width and height are reduced by half, but key information is still preserved. Since the hierarchical features of this other encoding pathway are obtained from pooling layers, these features are typically high-level features resulting from progressively reducing resolution through multiple pooling layers. They preserve global information in the image and provide crucial information for subsequent feature fusion.

[0059] Based on dimensional alignment, image semantic features and hierarchical features are fused at each encoding level through element-wise addition. The purpose of fusion is to combine the detailed semantic information of the image with the global information in the hierarchical features to obtain a richer feature representation. Through element-wise addition, the two can complement each other, so that the fused features contain both image detail information and contextual information of the global structure.

[0060] S140. The cross-modal features are fused with the encoded features to obtain decoder input features, and the target image is generated by the decoder based on the decoder input features;

[0061] Cross-modal features are fused with encoded features, and the decoder generates the target image based on the fused features from the input. The decoder is typically an upsampled network, such as a deconvolutional network, whose purpose is to generate a complete image from low-dimensional feature vectors. The generated image should accurately reflect the details of the medical image and conform to the content mentioned in the text description. In practical implementations, the decoder network design may include multiple convolutional layers, deconvolutional layers, and skip connections to ensure that the generated image has high-quality details.

[0062] Further, referring to Figure 5, step S140 includes:

[0063] S141. Project the cross-modal features into a spatial feature map through a fully connected layer;

[0064] S142. Concatenate the spatial feature map with the fused encoded feature to obtain the decoder input feature;

[0065] S143. The final upsampled image is obtained by progressively upsampling the input features of the decoder through a transposed convolutional network.

[0066] S144. The final upsampled image is mapped to the target medical image through an image-to-grayscale convolutional layer.

[0067] Through fully connected layer mapping, cross-modal features are upsampled and projected into a high-dimensional feature space using a multilayer perceptron (MLP) or transposed convolution, thus transforming into a spatial feature map. During the projection process, each element of the cross-modal features is weighted and summed, and then nonlinearly transformed using an activation function to obtain a spatial feature map suitable for subsequent processing. After projection, dimension matching is required to ensure that the projected feature map has the same size as the fused encoded feature space output by the dual encoder.

[0068] The spatial feature map and fused encoded features are concatenated along the channel dimension to generate decoder input features for a predefined specified channel, ready for subsequent image generation. Channel concatenation preserves dual-channel encoded features and cross-modal features, avoiding feature loss caused by single-modal dominance. Alternatively, element-wise multiplication can be used to merge the spatial feature map and encoded features.

[0069] The decoder upsamples the input features obtained above through a transposed convolutional network, gradually restoring the feature map size to the target image size, resulting in the final upsampled image. By using the transposed convolution operation, the feature map can be progressively restored from low resolution to higher resolution, preserving the image's structure and details. This process retains the main features of the image during restoration while reducing distortion.

[0070] The decoder then uses an image-to-grayscale convolutional layer to map the final upsampled image to the target medical image. This convolutional layer typically consists of several convolutional kernels, which, after convolution, transform the feature map into a grayscale or pseudo-color image of the medical image. This is to convert the high-dimensional feature map generated by the decoder into a format that meets the requirements of actual medical images and restore the visual details and semantic information of the original image. Through this mapping process, the decoder can ultimately generate a target image that meets medical standards, ensuring its realism and detail in grayscale and texture, while maintaining consistency with the type of the input medical image.

[0071] S150. Using the target image, construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the dual-channel encoder, and the decoder.

[0072] A joint loss function is constructed using the target image, which guides the model's optimization during training. The joint loss function is typically a combination of several individual loss functions, each focusing on a different optimization objective.

[0073] Further, referring to Figure 6, step S150 includes:

[0074] S151. Construct a joint loss function using the target image, wherein the joint loss function includes contrast loss and structural similarity loss, the contrast loss includes image mapping network loss and text mapping network loss, and the structural similarity loss includes luminance component loss, contrast component loss and structural component loss.

[0075] S152. Based on the joint loss function, the network parameters of the image encoder, the text encoder, the dual encoder, and the decoder are jointly optimized using the backpropagation algorithm.

[0076] The joint loss function includes contrastive loss and structural similarity loss. Contrastive loss measures the similarity between two modalities, ensuring that images and text are as consistent as possible in the feature space.

[0077] Image mapping network loss and text mapping network loss are two parts of the contrastive loss. Image mapping network loss measures the difference between the features of the generated image and the features of the target image, aiming to minimize the bias in the image feature space. Text mapping network loss measures the difference between the features of the text description and the features of the target image, ensuring consistency between the text information and the image content.

[0078] Structural similarity loss includes a luminance component loss to measure differences in image brightness, a contrast component loss to measure differences in image contrast, and a structural component loss to measure differences in structural information between images. The combination of these loss functions ensures that the optimization process not only considers intermodal consistency but also guarantees that the quality and structural features of the generated image closely approximate the target image.

[0079] By constructing such a joint loss function, it is possible to simultaneously optimize the matching degree between images and text and the quality of the images themselves during training. Especially in cross-modal generation tasks, the joint loss can effectively balance the feature fusion of different modalities, thereby improving the quality and accuracy of image generation.

[0080] The joint loss function serves as the optimization objective, and gradient calculation and parameter updates are performed using the backpropagation algorithm. Backpropagation updates the network parameters layer by layer by propagating the gradient of the loss function from the output layer back to the input layer. The backpropagation process involves the influence of each parameter on the loss function, effectively guiding the network to adjust towards minimizing the loss. By calculating the gradient of the joint loss function, the backpropagation algorithm simultaneously updates the parameters of the image encoder, text encoder, dual-channel encoder, and decoder, thereby achieving synchronous optimization of cross-modal information. This joint optimization not only improves the cross-modal matching capability of images and text but also ensures more accurate details in the image generation process. This joint optimization efficiently integrates features from multiple modules, improving the overall network performance and generation quality.

[0081] Furthermore, in addition to contrastive loss and structural similarity loss, perceptual loss or adversarial loss can also be considered. Perceptual loss measures the similarity between the generated image and the target image by the similarity of high-level features, which can enhance the perceptual quality of the image. Adversarial loss can enhance the realism of the image through generative adversarial networks (GANs), making the generated image more realistic.

[0082] Compared to existing technologies, the technical solution in this application extracts visual features from medical images and semantic features from text descriptions using an image encoder and a text encoder, respectively. Contrastive learning is employed to map the two types of features to a shared semantic space to achieve cross-modal alignment. A dual-path encoder is constructed to perform multi-scale feature extraction on the medical images: the first path captures hierarchical semantic information through a convolutional neural network, while the second path retains local salient features through max pooling. The outputs of the two paths are fused layer by layer to construct spatial context features. A multilayer perceptron projects the cross-modal features into a spatial feature map, which is then concatenated with the encoded features and input into the decoder for upsampling reconstruction. Contrastive loss and structural similarity loss are jointly optimized to achieve end-to-end training. Ultimately, this improves the SSIM (Spatial Similarity Index) of the generated medical images.

[0083] To more clearly illustrate the method of this application, the following will use a case study of using medical images of a rare hypertrophic osteoarthritis with a low incidence rate to train a deep learning model for detailed explanation.

[0084] Taking a medical image of hypertrophic osteoarthritis as an example, consider a grayscale CT image of the affected bone joint with dimensions of 224×244×1 and 1 channel, with the corresponding text "memorial phalanx periosteal hyperplasia". Preprocessing is performed on the image and text, specifically the image x... img Standardize the data to ensure that the numerical range of the data is consistent.

[0085] Perform image-text embedding. Input a grayscale CT image of the affected bone or joint. and the corresponding text description of the disease xtext ∈R L L represents the length of the input text.

[0086] For image x img Features are extracted using a pre-trained ResNet-50 network to obtain the image embedding z. img ,Right now, d i =2048 represents the dimension of the image embedding. For the textual description of the symptom x text The text embedding z is generated by processing the data using a pre-trained BERT model. text ,Right now, d t =768 represents the dimension of the text after word segmentation, taking BERT-base as an example.

[0087] Perform bimodal information fusion. After obtaining the image and text embeddings, z... img and z text Two mapping networks are trained through contrastive learning to map image and text information of different dimensions to the same shared space, with a dimension of d, such as 512 or 1024. Taking d=1024 as an example.

[0088] Image mapping network receiving Through a linear transformation The (linear transformation matrix) maps it to a vector z' in the shared space. img z′ img =Normalize(W img ·z img )

[0089] Normalize is used for normalization. For example, using L2 normalization ensures that the output vector has a unit length.

[0090] Text mapping network reception Through a linear transformation The (linear transformation matrix) maps it to a vector z' in the shared space. text z′ text =Normalize(W text ·z text )

[0091] Normalize is also a normalization method, such as using L2 normalization.

[0092] After obtaining image and text vectors of the same dimension, they are directly added together to fuse the bimodal information and obtain the final input.

[0093] The original image is used as the input to the encoder in the dual-path encoder-decoder. One path extracts image information through CNN, while the other path directly uses max pooling to downsample the image step by step to preserve local salient features.

[0094] Initial: The input is

[0095] This is information at level 0.

[0096] The CNN pathway uses 3×3 convolutional kernels with a stride of 2 and padding of 1. Simultaneously, as the network depth increases, the number of channels gradually increases to capture richer feature information. The dimensionality of the intermediate results after each CNN layer changes as follows: 224×224×1 → 112×112×32 → 56×56×64 → 28×28×128 → 14×14×256 → 7×7×512.

[0097] conv represents the result of convolution. Each layer uses ReLU as the activation function, and batch normalization (BN) is added to accelerate network training and improve stability.

[0098] The direct downsampling path processes the image through a max pooling layer with a stride of 2 and a window size of 2.

[0099] The downsampling process is as follows: 224×224×1 → 112×112×1 → 56×56×1 → 28×28×1 → 14×14×1 → 7×7×1. In each step, the information extracted by the CNN and the downsampled information are added element-wise before the next round of downsampling. The formula for each layer is illustrated below:

[0100] Overall calculation for each floor:

[0101] The final output is This refers to adding elements one by one. The last dimension was expanded to make the last dimension equal to... same.

[0102] The input features are used to generate the target image via a decoder. The low-resolution result from the dual-encoder is then used to... add The result of fusion of image and text through contrastive learning Add element by element The input x to the decoder is obtained by fusion. decode It is necessary to Remapped to the same dimension. This can be done using a multilayer perceptron (MLP). Mapped to 7×7×512.

[0103] Will The process involves progressively upsampling to the target resolution to restore details in medical images. Low-resolution feature maps are gradually upsampled to the target resolution using transposed convolutions: 7×7×512 → 14×14×512 → 28×28×512 → 56×56×512 → 112×112×512 → 224×224×512. Each layer uses a transposed convolution with a 2×2 kernel and a stride of 2 for upsampling, while batch normalization (BN) and ReLU are used as activation functions.

[0104] get Finally, a convolutional layer is added to map the image back to grayscale.

[0105] Ultimately, the generated grayscale CT images of hypertrophic osteoarthritis can be used as expanded data to train other deep learning models.

[0106] The target image is used to construct a joint loss function to jointly optimize the image encoder. During the training of the image-network mapping network, a contrastive loss function is used as the training objective. For each image i, its cosine similarity with text j is calculated. The larger the value, the more relevant the image is, and vice versa.

[0107] The loss of the image mapping network is L img The text mapping network loss is L text The total loss is

[0108] The codec generates images for training. The generated images are then processed. and target image y img Calculate the difference in brightness Among them, the image mean is used as the brightness. H and W represent the image's length and width:

[0109] Contrast difference Among them, image variance is used for contrast.

[0110] Structural differences

[0111] The differences in these three comparisons are taken as the loss.

[0112] The codec generates images for training.

[0113] Figure 7 is a schematic block diagram of a medical image generation device 600 based on bimodal fusion according to an embodiment of this application. As shown in Figure 7, corresponding to the above-described medical image generation method based on bimodal fusion, this application also provides a medical image generation device 600 based on bimodal fusion. This medical image generation device 600 based on bimodal fusion includes a unit for executing the above-described medical image generation method based on bimodal fusion, and the device can be configured in a terminal such as a desktop computer, tablet computer, or smartphone.

[0114] Specifically, referring to Figure 7, the medical image generation device 600 based on dual-modal fusion includes:

[0115] The information preprocessing unit 601 is used to acquire medical images and corresponding text descriptions, and perform standardized preprocessing to obtain preprocessed images and text sequences.

[0116] The cross-modal coding fusion unit 602 is used to extract image features from the preprocessed image through an image encoder, extract text features from the text sequence through a text encoder, and map the image features and the text features to a shared semantic space through contrastive learning and fuse them to obtain cross-modal features;

[0117] The dual-channel encoder working unit 603 is used to extract multi-scale features from the medical image through the dual-channel encoder to obtain fused encoded features;

[0118] Image decoding unit 604 is used to generate decoder input features by fusing the cross-modal features with the encoded features, and to generate a target image through a decoder based on the decoder input features;

[0119] The optimization training unit 605 is used to construct a joint loss function using the target image to jointly optimize the network parameters of the image encoder, the text encoder, the dual encoder, and the decoder.

[0120] Furthermore, the information preprocessing unit 601 includes:

[0121] The information acquisition unit 6011 is used to acquire the medical image and the corresponding text description through an interactive platform;

[0122] The image preprocessing unit 6012 is used to perform format standardization, numerical standardization and size unification processing on the medical image to obtain the preprocessed image;

[0123] The text preprocessing unit 6013 is used to extract key information, segment words, and normalize the length of the text description to obtain the text sequence.

[0124] Furthermore, the cross-modal coding fusion unit 602 includes:

[0125] Image encoding unit 6021 is used to input the preprocessed image into the image encoder for multi-level feature extraction to obtain an image embedding vector;

[0126] Text encoding unit 6022 is used to input the text sequence into the text encoder for contextual semantic encoding to obtain a text embedding vector.

[0127] Embedded vector mapping unit 6023 is used to map the image embedding vector and the text embedding vector to a shared semantic space through a mapping network trained by contrastive learning;

[0128] Embedding vector normalization unit 6024 is used to normalize the image embedding vector and the text embedding vector to obtain normalized image features and normalized text features;

[0129] The embedded vector fusion unit 6025 is used to perform element-wise addition and fusion of the normalized image features and the normalized text features to obtain cross-modal features.

[0130] Furthermore, the dual-channel encoder working unit 603 includes:

[0131] A dual-channel encoder input unit 6031 is used to take the medical image as the input of the dual-channel encoder;

[0132] The first encoder unit 6032 is used to extract image semantic features layer by layer through the first encoding path of the dual encoder using a convolutional neural network.

[0133] The second encoder unit 6033 is used to obtain hierarchical features by downsampling through the second encoding path of the dual encoder through the max pooling layer.

[0134] The fusion coding feature unit 6034 is used to add the image semantic features of the current level to the features of the level element by element at each coding level to obtain the fusion coding feature.

[0135] Furthermore, the image decoding unit 604 includes:

[0136] The fully connected layer projection unit 6041 is used to project the cross-modal features into a spatial feature map through the fully connected layer;

[0137] The channel splicing unit 6042 splices the spatial feature map with the fused encoded feature to obtain the decoder input feature;

[0138] The stepwise upsampling unit 6043 performs stepwise upsampling of the decoder input features through a transposed convolutional network to obtain the final upsampled image.

[0139] The image state restoration unit 6064 maps the final upsampled image to the target medical image through an image mapping-back-to-grayscale convolutional layer.

[0140] Furthermore, the optimization of training unit 605 includes:

[0141] The joint loss function establishment unit 6051 is used to construct a joint loss function through the target image, wherein the joint loss function includes contrast loss and structural similarity loss, the contrast loss includes image mapping network loss and text mapping network loss, and the structural similarity loss includes luminance component loss, contrast component loss and structural component loss;

[0142] The model optimization unit 6052 is used to jointly optimize the network parameters of the image encoder, the text encoder, the dual encoder, and the decoder using the backpropagation algorithm based on the joint loss function.

[0143] The aforementioned medical image generation device 600 based on dual-modal fusion can be implemented as a computer program that can run on the computer device shown in Figure 8.

[0144] Please refer to Figure 8, which is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a desktop computer, tablet computer, or smartphone. The server can be a standalone server or a server cluster composed of multiple servers.

[0145] Referring to Figure 8, the computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0146] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a medical image generation method based on bimodal fusion.

[0147] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0148] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a medical image generation method based on bimodal fusion.

[0149] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the structure shown in FIG8 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0150] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of the above method.

[0151] It should be understood that, in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0152] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium can be either non-volatile or volatile. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0153] Therefore, this application also provides a non-volatile computer-readable storage medium. This non-volatile computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the steps of the above-described method.

[0154] The non-volatile computer-readable storage medium can be any non-volatile computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0155] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0157] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0159] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A medical image generation method based on dual-modal fusion, comprising: Medical images and corresponding text descriptions are acquired, and standardized preprocessing is performed to obtain preprocessed images and text sequences. Image features are extracted from the preprocessed images using an image encoder, and text features are extracted from the text sequences using a text encoder. The image features and text features are mapped to a shared semantic space through contrastive learning and fused to obtain cross-modal features. The medical image is subjected to multi-scale feature extraction using a dual-channel encoder to obtain fused coded features; The cross-modal features are fused with the encoded features to obtain the decoder input features, and the target image is generated by the decoder based on the decoder input features; The target image is used to construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the dual encoder, and the decoder.

2. The medical image generation method based on dual-modal fusion according to claim 1, wherein, The steps of extracting image features from the preprocessed image using an image encoder and extracting text features from the text sequence using a text encoder include: The preprocessed image is input into the image encoder for multi-level feature extraction to obtain an image embedding vector; the text sequence is input into the text encoder for contextual semantic encoding to obtain a text embedding vector.

3. The medical image generation method based on dual-modal fusion according to claim 2, wherein, The step of mapping the image features and text features to a shared semantic space through contrastive learning and fusing them to obtain cross-modal features includes: The image embedding vector and the text embedding vector are mapped to a shared semantic space by a mapping network trained through contrastive learning. The image embedding vector and the text embedding vector are normalized to obtain normalized image features and normalized text features; The normalized image features and the normalized text features are fused element-wise to obtain cross-modal features.

4. The medical image generation method based on dual-modal fusion according to claim 3, wherein, The step of extracting multi-scale features from the medical image using a dual-channel encoder to obtain fused coded features includes: The medical image is used as the input to the dual-channel encoder; The first encoding path of the dual encoder uses a convolutional neural network to extract image semantic features layer by layer; the second encoding path of the dual encoder uses max pooling layers to downsample layer by layer to obtain layer features; at each encoding layer, the image semantic features of the current layer are added to the layer features element by element to obtain fused encoding features.

5. The medical image generation method based on dual-modal fusion according to claim 4, wherein, The step of fusing the cross-modal features with the encoded features to obtain the decoder input features includes: The cross-modal features are projected into a spatial feature map through a fully connected layer; The spatial feature map and the fused encoded feature are concatenated by channels to obtain the decoder input feature.

6. The medical image generation method based on dual-modal fusion according to claim 5, wherein, The step of generating the target image through the decoder based on the decoder input features includes: The decoder input features are upsampled step by step through a transposed convolutional network to obtain the final upsampled image; the final upsampled image is then mapped to the target medical image through an image-to-grayscale convolutional layer.

7. The medical image generation method based on dual-modal fusion according to claim 6, wherein, The step of using the target image to construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the dual-channel encoder, and the decoder includes: A joint loss function is constructed using the target image, wherein the joint loss function includes contrast loss and structural similarity loss, the contrast loss includes image mapping network loss and text mapping network loss, and the structural similarity loss includes luminance component loss, contrast component loss and structural component loss; Based on the joint loss function, the network parameters of the image encoder, the text encoder, the dual encoder, and the decoder are jointly optimized using the backpropagation algorithm.

8. A medical image generation device based on dual-modal fusion, comprising: The information preprocessing unit is used to acquire medical images and corresponding text descriptions, and perform standardized preprocessing to obtain preprocessed images and text sequences. The cross-modal coding fusion unit is used to extract image features from the preprocessed image through an image encoder, extract text features from the text sequence through a text encoder, and map the image features and the text features to a shared semantic space through contrastive learning and fuse them to obtain cross-modal features; The dual-channel encoder working unit is used to extract multi-scale features from the medical image through the dual-channel encoder to obtain fused encoded features; An image decoding unit is used to generate decoder input features by fusing the cross-modal features with the encoded features, and to generate a target image through a decoder based on the decoder input features. An optimization training unit is used to construct a joint loss function using the target image to jointly optimize the network parameters of the image encoder, the text encoder, the dual-channel encoder, and the decoder.

9. A computer device, comprising a memory and a processor connected to the memory; the memory being used to store a computer program; wherein, The processor is used to run a computer program stored in the memory to perform the following steps: Medical images and corresponding text descriptions are acquired, and standardized preprocessing is performed to obtain preprocessed images and text sequences. Image features are extracted from the preprocessed images using an image encoder, and text features are extracted from the text sequences using a text encoder. The image features and text features are mapped to a shared semantic space through contrastive learning and fused to obtain cross-modal features. The medical image is subjected to multi-scale feature extraction using a dual-channel encoder to obtain fused coded features; The cross-modal features are fused with the encoded features to obtain the decoder input features, and the target image is generated by the decoder based on the decoder input features; The target image is used to construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the dual encoder, and the decoder.

10. The computer device according to claim 9, wherein, The step of extracting image features from the preprocessed image using an image encoder and extracting text features from the text sequence using a text encoder includes: The preprocessed image is input into the image encoder for multi-level feature extraction to obtain the image embedding vector; The text sequence is input into the text encoder for contextual semantic encoding to obtain a text embedding vector.

11. The computer device according to claim 10, wherein, The step of mapping the image features and text features to a shared semantic space through contrastive learning and fusing them to obtain cross-modal features includes: The image embedding vector and the text embedding vector are mapped to a shared semantic space by a mapping network trained through contrastive learning. The image embedding vector and the text embedding vector are normalized to obtain normalized image features and normalized text features; The normalized image features and the normalized text features are fused element-wise to obtain cross-modal features.

12. The computer device according to claim 11, wherein, The step of extracting fused coded features from the medical image using a dual-channel encoder at multiple scales includes: The medical image is used as the input to the dual-channel encoder; The first encoding path of the dual encoder uses a convolutional neural network to extract image semantic features layer by layer; the second encoding path of the dual encoder uses max pooling layers to downsample layer by layer to obtain layer features; at each encoding layer, the image semantic features of the current layer are added to the layer features element by element to obtain fused encoding features.

13. The computer device according to claim 12, wherein, The process of fusing the cross-modal features with the encoded features to obtain the decoder input features includes: The cross-modal features are projected into a spatial feature map through a fully connected layer; The spatial feature map and the fused encoded feature are concatenated by channels to obtain the decoder input feature.

14. The computer device according to claim 13, wherein, The step of generating the target image through the decoder based on the decoder input features includes: The decoder input features are upsampled step by step through a transposed convolutional network to obtain the final upsampled image; the final upsampled image is then mapped to the target medical image through an image-to-grayscale convolutional layer.

15. The computer device according to claim 14, wherein, The step of using the target image to construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the dual-channel encoder, and the decoder includes: A joint loss function is constructed using the target image, wherein the joint loss function includes contrast loss and structural similarity loss, the contrast loss includes image mapping network loss and text mapping network loss, and the structural similarity loss includes luminance component loss, contrast component loss and structural component loss; Based on the joint loss function, the network parameters of the image encoder, the text encoder, the dual encoder, and the decoder are jointly optimized using the backpropagation algorithm.

16. A non-volatile computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the following steps: Medical images and corresponding text descriptions are acquired, and standardized preprocessing is performed to obtain preprocessed images and text sequences. Image features are extracted from the preprocessed images using an image encoder, and text features are extracted from the text sequences using a text encoder. The image features and text features are mapped to a shared semantic space through contrastive learning and fused to obtain cross-modal features. The medical image is subjected to multi-scale feature extraction using a dual-channel encoder to obtain fused coded features; The cross-modal features are fused with the encoded features to obtain the decoder input features, and the target image is generated by the decoder based on the decoder input features; The target image is used to construct a joint loss function to jointly optimize the network parameters of the image encoder, the text encoder, the dual encoder, and the decoder.

17. The non-volatile computer-readable storage medium according to claim 16, wherein, The step of extracting image features from the preprocessed image using an image encoder and extracting text features from the text sequence using a text encoder includes: The preprocessed image is input into the image encoder for multi-level feature extraction to obtain an image embedding vector; the text sequence is input into the text encoder for contextual semantic encoding to obtain a text embedding vector.

18. The non-volatile computer-readable storage medium according to claim 17, wherein, The step of mapping the image features and text features to a shared semantic space through contrastive learning and fusing them to obtain cross-modal features includes: The image embedding vector and the text embedding vector are mapped to a shared semantic space by a mapping network trained through contrastive learning. The image embedding vector and the text embedding vector are normalized to obtain normalized image features and normalized text features; The normalized image features and the normalized text features are fused element-wise to obtain cross-modal features.

19. The non-volatile computer-readable storage medium according to claim 18, wherein, The step of extracting fused coded features from the medical image using a dual-channel encoder at multiple scales includes: The medical image is used as the input to the dual-channel encoder; The first encoding path of the dual encoder uses a convolutional neural network to extract semantic features of the image layer by layer. The hierarchical features are obtained by downsampling through the second encoding path of the dual encoder using a max pooling layer. At each coding level, the semantic features of the image at the current level are added element-wise to the features at the same level to obtain the fused coding features.

20. The non-volatile computer-readable storage medium according to claim 19, wherein, The process of fusing the cross-modal features with the encoded features to obtain the decoder input features includes: The cross-modal features are projected into a spatial feature map through a fully connected layer; The spatial feature map and the fused encoded feature are concatenated by channels to obtain the decoder input feature.

Images