Multi-parameter MRI-based breast cancer auxiliary decision-making method, and related apparatus

Abstract

Disclosed in the present invention are a multi-parameter MRI-based breast cancer auxiliary decision-making method, and a related apparatus. The method comprises: acquiring an MRI dataset, comprising MRI sequences of different modalities; performing feature extraction on multi-parameter MRI of sample data to obtain multi-modal features; using the multi-modal features and label data to perform optimization training on a preset model, so as to obtain a target model, in the preset model, a sparse mixture-of-experts being added to a preceding Transformer block, and a soft mixture-of-experts model being added to a following Transformer block; and using the target model to perform breast cancer auxiliary decision-making. The present invention can learn specific feature representations for different MRI modalities, can implement deep fusion and interactive modeling of multi-modal information, effectively improves the accuracy and robustness of models with respect to differential diagnosis of breast cancer, lays a technical foundation for non-invasive and personalized management, and can be widely used in the technical field of imaging data processing.

Description

A method and related equipment for breast cancer auxiliary decision-making based on multiparameter MRI Technical Field

[0001] This invention relates to the field of imaging data processing technology, and in particular to a method and related equipment for breast cancer auxiliary decision-making based on multi-parameter MRI. Background Technology

[0002] Breast cancer is one of the most common malignant tumors, and early, accurate diagnosis and personalized treatment are crucial for improving patient prognosis. Currently, breast magnetic resonance imaging (MRI) has become a key imaging modality for breast cancer detection and evaluation due to its high soft tissue resolution and high sensitivity to changes in blood supply.

[0003] With the development of artificial intelligence technology, deep learning models have been widely used in medical image analysis to assist in the diagnosis and assessment of breast cancer. Existing technologies are usually based on single-modality MRI sequences (such as dynamic contrast-enhanced MRI) for model training. Although they can identify malignant tumors to some extent, they lack specificity and fail to fully integrate the rich complementary information contained in multi-parameter MRI, thus limiting the upper limit of model performance.

[0004] Furthermore, effectively fusing heterogeneous multimodal medical image data is a significant technical challenge. Traditional multimodal fusion methods (such as simple feature stitching or post-decision fusion) often struggle to model the complex and deep-seated interactions between modalities, resulting in limited fusion effectiveness. Meanwhile, while existing general-purpose visual models possess powerful representational capabilities, their massive parameter scale and typical training based on natural images present significant domain gaps when directly transferred to the higher-dimensional, more complex 3D multiparameter medical image domain. This leads to poor model performance, low training efficiency, and difficulty in handling common clinical scenarios involving missing modalities. Summary of the Invention

[0005] The main objective of this invention is to provide a method, device, electronic device, storage medium, and program product for breast cancer auxiliary decision-making based on multi-parameter MRI, aiming to solve at least one problem of the prior art.

[0006] To achieve the above objectives, one aspect of this invention proposes a breast cancer-assisted decision-making method based on multi-parameter MRI, the method comprising:

[0007] Obtain the MRI dataset; where each sample in the MRI dataset includes multi-parameter MRI and label data of the same breast sample, and the multi-parameter MRI includes MRI sequences of different modalities;

[0008] Multi-parameter MRI features were extracted from the sample data to obtain multimodal features;

[0009] The preset model is optimized and trained using multimodal features and label data to obtain the target model;

[0010] The preset model includes a preset number of Transformer blocks. In the preset model, a sparse modal expert mixture is added to the first number of Transformer blocks, and a soft expert model is added to the second number of Transformer blocks. The sum of the first number and the second number is the preset number. The sparse modal expert mixtures in different Transformer blocks are configured to process MRI sequences from different modalities within the same Transformer block.

[0011] Using target models for breast cancer decision support.

[0012] In some embodiments, multiparametric MRI includes T2-weighted imaging, diffusion-weighted imaging, and dynamic contrast-enhanced imaging. Feature extraction of multiparametric MRI data from sample data to obtain multimodal features includes the following steps:

[0013] The embedding operation of MRI sequences is performed using a tokenizer to obtain the embedding features corresponding to each modality of MRI sequence;

[0014] The embedded features are flattened in dimension, and then a preset CLS token and position embedding are added to obtain the input features;

[0015] Input features corresponding to different modal MRI sequences are concatenated to form multimodal features.

[0016] In some embodiments, the tokenizer includes multiple convolutional layers and a max-pooling layer. The tokenizer is used to embed MRI sequences, including the following steps:

[0017] The height, width, and number of slices of the MRI sequence are downsampled by a preset factor through multiple convolutional layers and max pooling layers;

[0018] The preset multiplier is determined based on the product of the stride of all convolutional layers and the stride of the max pooling layer.

[0019] In some embodiments, the method further includes the following steps:

[0020] Obtain a general multimodal base model; wherein the general multimodal base model includes the preset number of Transformer blocks;

[0021] The sparse modal expert hybrid is added to the first number of Transformer blocks before the general multimodal base model, and the soft expert model is added to the second number of Transformer blocks after the general multimodal base model to construct the initial model;

[0022] The sparse modal expert hybrid contains multiple parallel sparse expert networks configured for different modalities within each Transformer block;

[0023] The pre-trained parameters of the general multimodal base model are loaded into the corresponding shared layer in the initial model, thereby initializing the parameters of the sparse modal expert hybrid and the soft expert model, and configuring them to obtain the preset model;

[0024] In the preset model, the first number of Transformer blocks are configured to encode the MRI sequences of different modalities within a single block using different sparse expert networks.

[0025] In some embodiments, a CLS token is appended to the multimodal features. The preset model is optimized and trained using the multimodal features and label data to obtain the target model, including the following steps:

[0026] The multimodal features are input into a preset model, processed by a preset number of Transformer blocks, and the CLS token corresponding to the CLS token is extracted.

[0027] The CLS labels are subjected to layer normalization and linear classification to obtain the training inference results;

[0028] A loss function is constructed based on the training inference results and labeled data, and then the parameters of the preset model are tuned and optimized to obtain the target model.

[0029] In some embodiments, using a target model for breast cancer-assisted decision-making includes the following steps:

[0030] Acquire the true multiparametric MRI images to be processed;

[0031] Feature extraction is performed on real multiparameter MRI to obtain real multimodal features as the first input data; wherein, the real multimodal features are appended with CLS tokens;

[0032] The first Transformer block in the target model is used as the first processing block;

[0033] The first input data is input into the first processing block, and the first output data is obtained by processing based on the sparse modal expert hybrid.

[0034] The next Transformer block is used as the first processing block, the first output data is used as the first input data, and the process of inputting the first input data into the first processing block is repeated until the number of first processing blocks reaches the first number.

[0035] Use the next Transformer block as the second processing block, and use the first output data as the second input data;

[0036] The second input data is input into the second processing block, and the second output data is obtained by processing based on the soft expert model.

[0037] Use the next Transformer block as the second processing block, the second output data as the second input data, and return to execute the step of inputting the second input data into the second processing block until the number of processes passing through the second processing block reaches the second number.

[0038] The sum of the first quantity and the second quantity is the preset quantity;

[0039] Extract the CLS token corresponding to the CLS token from the second output data of the last Transformer block;

[0040] Layered normalization and linear classification were performed on the CLS markers to obtain predictive inference results, which served as the basis for auxiliary decision-making in breast cancer.

[0041] In some embodiments, the first processing block includes a multi-head self-attention mechanism, a feedforward network, layer normalization, and a sparse modality expert hybrid. The sparse modality expert hybrid contains multiple parallel sparse expert networks configured for different modalities within each Transformer block. The first input data is input into the first processing block, and the first output data is obtained by processing based on the sparse modality expert hybrid, including the following steps:

[0042] The first input data is processed sequentially through layer normalization and multi-head self-attention mechanism to obtain the first intermediate data;

[0043] Add the first intermediate data to the first input data to obtain the second intermediate data;

[0044] The second intermediate data is passed sequentially through the layer normalization and the feedforward network to obtain the third intermediate data;

[0045] The second intermediate data is subjected to layer normalization, and the results of the layer normalization are divided into multiple modal subsequences according to the modal source;

[0046] Each of the modal subsequences is input into the corresponding sparse expert network for feature mapping to obtain multiple expert output sequences.

[0047] The multiple expert output sequences are concatenated along the sequence dimension to obtain the fourth intermediate data;

[0048] The expression for the sparse modal expert hybrid is: MOME Sparse (X)=LN(LP(GELU(LP(X)))),

[0049] In the formula, X represents the input of the sparse modal expert hybrid; MOME Sparse MOME represents a sparse modal expert hybrid. Sparse (X) represents the output of the sparse modal expert mixture; LN represents layer normalization; LP represents the linear projection layer; GELU represents the Gaussian error linear unit;

[0050] The second, third, and fourth intermediate data are added together to obtain the first output data.

[0051] In some embodiments, the second processing block includes a multi-head self-attention mechanism, a feedforward network, layer normalization, and a soft expert model. The second input data is input into the second processing block, and the second output data is obtained based on the soft expert model, including the following steps:

[0052] The second input data is processed sequentially through layer normalization and multi-head self-attention mechanism to obtain the fifth intermediate data;

[0053] Add the fifth intermediate data to the second input data to obtain the sixth intermediate data;

[0054] The sixth intermediate data is passed through layer normalization and feedforward network in sequence to obtain the seventh intermediate data;

[0055] The sixth intermediate data is then processed sequentially through layer normalization and a soft expert model to obtain the eighth intermediate data.

[0056] The expression for the soft expert model is: MOME Soft (X)=LN(LP(SMoE(GELU(LP(X))))),

[0057] In the formula, X represents the input of the soft expert model; MOME Soft This represents the soft expert model; MOME Soft (X) represents the output of the soft expert model; LN represents layer normalization; LP represents the linear projection layer; SMoE represents the soft expert mixture; GELU represents the Gaussian error linear unit.

[0058] Add the sixth, seventh, and eighth intermediate data to obtain the second output data.

[0059] To achieve the above objectives, another aspect of the present invention provides a breast cancer auxiliary decision-making device based on multi-parameter MRI, the device comprising:

[0060] The first module is used to acquire the MRI dataset; in which each sample data in the MRI dataset includes multi-parameter MRI and label data of the same breast sample, and the multi-parameter MRI includes MRI sequences of different modalities;

[0061] The second module is used to extract features from the multi-parameter MRI of the sample data to obtain multimodal features;

[0062] The third module is used to optimize and train the preset model using multimodal features and label data to obtain the target model;

[0063] The preset model includes a preset number of Transformer blocks. In the preset model, a sparse modal expert mixture is added to the first number of Transformer blocks, and a soft expert model is added to the second number of Transformer blocks. The sum of the first number and the second number is the preset number. The sparse modal expert mixtures in different Transformer blocks are configured to process MRI sequences from different modalities within the same Transformer block.

[0064] The fourth module is used to assist in decision-making for breast cancer using the target model.

[0065] In some embodiments, the apparatus further includes a fifth module for performing the following operations:

[0066] Obtain a general multimodal base model; wherein the general multimodal base model includes the preset number of Transformer blocks;

[0067] The sparse modal expert hybrid is added to the first number of Transformer blocks before the general multimodal base model, and the soft expert model is added to the second number of Transformer blocks after the general multimodal base model to construct the initial model;

[0068] The sparse modal expert hybrid contains multiple parallel sparse expert networks configured for different modalities within each Transformer block;

[0069] The pre-trained parameters of the general multimodal base model are loaded into the corresponding shared layer in the initial model, thereby initializing the parameters of the sparse modal expert hybrid and the soft expert model, and configuring them to obtain the preset model;

[0070] In the preset model, the first number of Transformer blocks are configured to encode the MRI sequences of different modalities within a single block using different sparse expert networks.

[0071] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0072] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0073] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0074] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a method, device, electronic device, storage medium, and program product for breast cancer assisted decision-making based on multi-parameter MRI. This solution acquires an MRI dataset; wherein each sample data in the MRI dataset includes multi-parameter MRI and label data of the same breast sample, and the multi-parameter MRI includes MRI sequences of different modalities; features are extracted from the multi-parameter MRI of the sample data to obtain multi-modal features; a preset model is optimized and trained using the multi-modal features and label data to obtain a target model; wherein the preset model includes a preset number of Transformer blocks, wherein: a sparse modality expert mixture is added to a first number of Transformer blocks, and a soft expert model is added to a second number of Transformer blocks, the sum of the first and second numbers being the preset number; the sparse modality expert mixtures in different Transformer blocks are configured to process MRI sequences from different modalities within the same Transformer block; and the target model is used for breast cancer assisted decision-making. This invention introduces a sparse modal expert hybrid in the preceding Transformer block, enabling the model to learn specific feature representations for different MRI modalities. Furthermore, it introduces a soft expert model in the subsequent Transformer block, achieving deep fusion and interactive modeling of multimodal information. This hierarchical and structured expert hybrid design overcomes the limitations of traditional methods in multimodal fusion, effectively improving the model's accuracy and robustness in the differential diagnosis of breast cancer, and laying a technical foundation for achieving non-invasive and personalized management. Attached Figure Description

[0075] Figure 1 is a schematic diagram of an implementation environment for a breast cancer auxiliary decision-making method based on multi-parameter MRI provided in an embodiment of the present invention;

[0076] Figure 2 is a flowchart illustrating a breast cancer assisted decision-making method based on multi-parameter MRI provided in an embodiment of the present invention;

[0077] Figure 3 is a schematic diagram of the overall application process of the breast cancer auxiliary decision-making method based on multi-parameter MRI provided in the embodiment of the present invention;

[0078] Figure 4 is a schematic diagram of an example of MOME decision interpretation provided in an embodiment of the present invention;

[0079] Figure 5 is a schematic diagram of a breast cancer auxiliary decision-making device based on multi-parameter MRI provided in an embodiment of the present invention;

[0080] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0082] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0083] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0084] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0085] In related technologies, traditional multimodal fusion methods (such as simple feature splicing or post-decision fusion) often struggle to model complex and deep-level interactions between modalities, resulting in limited fusion effects.

[0086] In view of this, this invention provides a method and related device for breast cancer assisted decision-making based on multi-parameter MRI. The method involves acquiring an MRI dataset; each sample in the MRI dataset includes multi-parameter MRI and label data of the same breast sample, and the multi-parameter MRI includes MRI sequences of different modalities; feature extraction is performed on the multi-parameter MRI of the sample data to obtain multi-modal features; a preset model is optimized and trained using the multi-modal features and label data to obtain a target model; the preset model includes a preset number of Transformer blocks, wherein: a sparse modality expert mixture is added to a first number of Transformer blocks, and a soft expert model is added to a second number of Transformer blocks, the sum of the first and second numbers being the preset number; the sparse modality expert mixtures in different Transformer blocks are configured to process MRI sequences from different modalities within the same Transformer block; and the target model is used for breast cancer assisted decision-making. This invention introduces a sparse modal expert hybrid in the preceding Transformer block, enabling the model to learn specific feature representations for different MRI modalities. Furthermore, it introduces a soft expert model in the subsequent Transformer block, achieving deep fusion and interactive modeling of multimodal information. This hierarchical and structured expert hybrid design overcomes the limitations of traditional methods in multimodal fusion, effectively improving the model's accuracy and robustness in the differential diagnosis of breast cancer, and laying a technical foundation for achieving non-invasive and personalized management.

[0087] It is understood that the breast cancer assisted decision-making method based on multi-parameter MRI provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, or desktop computer, but it is not limited to these.

[0088] Figure 1 illustrates an implementation environment according to an embodiment of the present invention. Referring to Figure 1, this implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0089] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0090] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0091] Terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0092] For example, based on the implementation environment shown in FIG1, this embodiment of the invention provides a breast cancer auxiliary decision-making method based on multi-parameter MRI. The following description takes the application of this breast cancer auxiliary decision-making method based on multi-parameter MRI in server 101 as an example. It can be understood that this breast cancer auxiliary decision-making method based on multi-parameter MRI can also be applied in terminal 102.

[0093] Referring to Figure 2, which is an optional flowchart of a breast cancer auxiliary decision-making method based on multiparameter MRI provided in an embodiment of the present invention, the executing entity of this breast cancer auxiliary decision-making method based on multiparameter MRI can be any of the aforementioned computer devices (including servers or terminals). The method in Figure 2 may include, but is not limited to, steps S100 to S400.

[0094] Step S100: Obtain the MRI dataset;

[0095] The MRI dataset includes multi-parameter MRI and label data for each sample of the same breast sample. The multi-parameter MRI includes MRI sequences of different modalities.

[0096] For example, in some specific implementations, MRI datasets can be compiled by collecting data from different hospitals and covering populations in different regions. Specifically, multiparameter MRI may include T2-weighted imaging, diffusion-weighted imaging, and dynamic contrast-enhanced imaging.

[0097] Step S200: Perform feature extraction on the multi-parameter MRI of the sample data to obtain multimodal features;

[0098] It should be noted that multiparameter MRI includes T2-weighted imaging, diffusion-weighted imaging, and dynamic contrast-enhanced imaging. In some embodiments, step S200 may include the following steps: using a tokenizer to embed the MRI sequence to obtain the embedding features corresponding to each modality of MRI sequence; flattening the embedding features in dimensions, and then attaching a preset CLS token and position embedding to obtain input features; and connecting the input features corresponding to different modalities of MRI sequence as multimodal features.

[0099] Specifically, in this embodiment of the invention, a tokenizer is used to transform a high-dimensional 3D image sequence into a standardized feature token sequence, and a CLS token is attached, providing a structured input for subsequent Transformer-based model processing, which can ensure the consistency of feature representation and the efficiency of model processing.

[0100] For example, in some specific implementations, each MRI sequence utilizes a structurally identical tokenizer module to embed the input as a token, i.e., a series of embedded features, denoted as X. DCE X DWI and X T2 (DCE, DWI, and T2 correspond to dynamic contrast-enhanced imaging, diffusion-weighted imaging, and T2-weighted imaging, respectively). The obtained feature maps are then flattened in width, height, and slice dimensions to form a sequence input. For example, the original shape is... The DCE-MRI will be processed by the tokenizer into And flattened into A CLS token Attached to X DCE The data is then processed and used for final classification. Finally, a learnable one-dimensional positional embedding is added to the input token. The multimodal features from different sequences are then concatenated and fed into the subsequent Transformer structure.

[0101] It should be noted that the tokenizer includes multiple convolutional layers and a max pooling layer. In some embodiments, the embedding operation of the MRI sequence using the tokenizer may include the following steps: downsampling the height, width and number of slices of the MRI sequence by a preset factor through multiple convolutional layers and a max pooling layer; wherein the preset factor is determined based on the product of the stride of all convolutional layers and the stride of the max pooling layer.

[0102] Specifically, the embodiments of the present invention can achieve efficient dimensionality reduction and feature compression of the original MRI data through multi-layer convolution and downsampling operations. This design significantly reduces the amount of computation input to the subsequent Transformer model, improves the processing speed of the entire system, and preserves key spatial and semantic information.

[0103] For example, in some implementations, the tokenizer comprises three 3D convolutional layers (stride = 2) and terminates with a max-pooling layer (stride = 2). Each convolutional layer is followed by an instance normalization layer and a ReLU layer. The tokenizer downsamples the width, height, and number of slices of the MRI input by a factor of 1 / 16 and generates a feature map with 768 feature dimensions. The resulting feature map is then flattened in the width, height, and slice dimensions to form the sequence input.

[0104] Step S300: Optimize and train the preset model using multimodal features and label data to obtain the target model;

[0105] The preset model includes a preset number of Transformer blocks. In the preset model, a sparse modal expert mixture is added to the first number of Transformer blocks, and a soft expert model is added to the second number of Transformer blocks. The sum of the first number and the second number is the preset number. The sparse modal expert mixtures in different Transformer blocks are configured to process MRI sequences from different modalities within the same Transformer block.

[0106] It should be noted that, in some embodiments, the method may further include the following steps: obtaining a general multimodal base model; wherein the general multimodal base model includes the preset number of Transformer blocks; adding the sparse modality expert hybrid to the first number of Transformer blocks preceding the general multimodal base model, and adding the soft expert model to the second number of Transformer blocks following the general multimodal base model to construct an initial model; wherein the sparse modality expert hybrid contains multiple parallel sparse expert networks configured for different modalities within each Transformer block; loading the pre-trained parameters of the general multimodal base model into the corresponding shared layer in the initial model, thereby initializing the parameters of the sparse modality expert hybrid and the soft expert model to configure the preset model; wherein the first number of Transformer blocks in the preset model are configured to encode the MRI sequences of different modalities within a single block using different sparse expert networks.

[0107] Specifically, this embodiment of the invention freezes most of the parameters of the base model and only fine-tunes the introduced expert hybrid module. This leverages the general representational capabilities learned by the base model on large datasets while allowing it to be quickly adapted to specific 3D multi-parameter medical imaging tasks using only a small number of trainable parameters. This method significantly reduces the dependence on the amount of labeled medical data, improves model development efficiency, and effectively avoids the problem of overfitting on small datasets due to excessive parameters.

[0108] It should be noted that the multimodal features are accompanied by CLS tokens. In some embodiments, step S300 may include the following steps: inputting the multimodal features into a preset model, processing them through a preset number of Transformer blocks, and extracting the CLS tokens corresponding to the CLS tokens; performing layer normalization and linear classification on the CLS tokens to obtain the training inference results; constructing a loss function based on the training inference results and the label data, and then tuning and optimizing the preset model to obtain the target model.

[0109] Specifically, this embodiment of the invention utilizes CLS tokens to aggregate global information from the entire sequence, and outputs decision results through layer normalization and a linear classifier, resulting in a clear and efficient process. Finally, a loss function is constructed and optimized based on the prediction results and the gold standard label, ensuring the correctness of the model's learning direction and ultimately obtaining a high-performance target model.

[0110] For example, in some implementations, MOME's Transformer structure is adapted from BEiT3. BEiT3 was originally a visual language foundation model with 276 million parameters, pre-trained from 21 million image-text pairs, 14 million images, and 160GB of documents. This model contains twelve structurally identical Transformer blocks for feature encoding, the process of which can be described as follows: Z l =X l +MSA((LN(X l )))(1) X l+1 =Z l +FFN(LN(Z l ))(2)

[0111] Among them, Z l Yes, it is the original input X. l Contextual information refined through self-attention mechanisms (MSA(LN(X)) l The fusion of ))), X l It is the input of the l-th block, MSA represents multi-head self-attention mechanism, FFN represents feedforward network with two linear projection layers, LN represents layer normalization, X l+1 It outputs and inputs into the (l+1)th Transformer block.

[0112] Based on the BEiT3 model, this invention further introduces a Modal Expert Hybrid (MOME) to achieve multimodal learning and fusion. Specifically, this invention fixes the pre-training parameters of BEiT3 and injects simple trainable modules. This invention sets the first k layers to learn from different modalities (i.e., different MRI sequences) by adding a sparse Modal Expert Hybrid to the Transformer blocks. Furthermore, the original BEiT3 does not have the ability to fuse multi-parameter MRI information. Therefore, multimodal fusion is accomplished by the last 12-k Transformer blocks using a soft MOME (i.e., a soft expert model).

[0113] Step S400: Use the target model to perform breast cancer auxiliary decision-making;

[0114] It should be noted that in some embodiments, step S400 may include the following steps: acquiring the real multi-parameter MRI to be processed; performing feature extraction on the real multi-parameter MRI to obtain real multi-modal features as the first input data; wherein, the real multi-modal features are appended with a CLS token; using the first Transformer block in the target model as the first processing block; inputting the first input data into the first processing block, and processing it based on the sparse modality expert hybrid to obtain the first output data; using the next Transformer block as the first processing block, using the first output data as the first input data, and returning to execute the step of inputting the first input data into the first processing block until the number of first processing blocks reaches a first number; and using the next Transformer block as the first processing block, and ..., until the number of first processing blocks reaches a first number; and so on. The Transformer block is used as the second processing block, taking the first output data as the second input data; the second input data is input into the second processing block, and processed based on the soft expert model to obtain the second output data; the next Transformer block is used as the second processing block, and the second output data is used as the second input data, returning to execute the step of inputting the second input data into the second processing block, until the number of second processing blocks reaches a second quantity; where the sum of the first quantity and the second quantity is a preset quantity; the CLS token corresponding to the CLS token is extracted from the second output data output by the last Transformer block; the CLS token is subjected to layer normalization and linear classification to obtain the prediction inference result as the basic data for breast cancer auxiliary decision-making.

[0115] The data flow remains a spliced ​​sequence throughout the model; specifically, this splicing is an output aggregation operation within the sparse expert module.

[0116] Specifically, this embodiment of the invention applies a sparse modality expert hybrid and a soft expert model in stages, and the inference process completely replicates the feature learning and fusion mechanism during training, ensuring the stable performance of the model. The design of this embodiment makes the model suitable not only for complete multi-parameter inputs, but its modular structure also makes it possible to handle common clinical situations where some modalities are missing. Simply remove the processing branch of the corresponding modality during inference, enhancing the model's practicality and flexibility.

[0117] It should be noted that the first processing block includes a multi-head self-attention mechanism, a feedforward network, layer normalization, and a sparse modality expert hybrid. The sparse modality expert hybrid contains multiple parallel sparse expert networks configured for different modalities within each Transformer block. In some embodiments, inputting the first input data into the first processing block and processing it based on the sparse modality expert hybrid to obtain the first output data may include the following steps: processing the first input data sequentially through layer normalization and the multi-head self-attention mechanism to obtain first intermediate data; adding the first intermediate data to the first input data to obtain second intermediate data; processing the second intermediate data sequentially through the layer normalization and the feedforward network to obtain third intermediate data; performing layer normalization on the second intermediate data and dividing the normalized data result into multiple modality sub-sequences according to the modality source; inputting each modality sub-sequence into the corresponding sparse expert network for feature mapping to obtain multiple expert output sequences; concatenating the multiple expert output sequences along the sequence dimension to obtain fourth intermediate data; and adding the second, third, and fourth intermediate data to obtain the first output data.

[0118] Specifically, the embodiments of the present invention clearly demonstrate how this module can inject modality-specific feature enhancements without interfering with the backbone network information flow by formulaically defining its integration method in the Transformer block (adding in parallel with MSA and FFN), thereby optimizing the extraction quality of single-modality features.

[0119] For example, in some specific embodiments, the present invention sets the first k layers to learn from different modalities (i.e., different MRI sequences) by adding a sparse modality expert hybrid to the Transformer block, the process of which can be represented as follows:

[0120] Where * indicates that the module parameters are loaded from the pre-trained BEiT3 and fixed during training, and i is the index. Specifically, It comes from set {X} DCE,1 ,X DWI,1 ,X T2,1 The i-th feature of} It is the i-th sparse MOME. Sparse A feedforward network structure employing two layers of linear projection followed by a normalized layer is used: MOME Sparse (X)=LN(LP(GELU(LP(X))))(5)

[0121] Where X represents the input of the sparse modal expert hybrid; MOME Sparse MOME represents a sparse modal expert hybrid. Sparse(X) represents the output of the sparse modal expert hybrid; LP represents the linear projection layer, and GELU represents the Gaussian error linear unit. In this way, each sparse MOME will learn to encode a specific type of input sequence, while most of the Transformer block parameters are fixed and shared by each modality.

[0122] It should be noted that the second processing block includes a multi-head self-attention mechanism, a feedforward network, layer normalization, and a soft expert model. In some embodiments, the second input data is input into the second processing block, and the second output data is obtained based on the soft expert model. This may include the following steps: processing the second input data sequentially through layer normalization and the multi-head self-attention mechanism to obtain fifth intermediate data; adding the fifth intermediate data to the second input data to obtain sixth intermediate data; processing the sixth intermediate data sequentially through layer normalization and the feedforward network to obtain seventh intermediate data; processing the sixth intermediate data sequentially through layer normalization and the soft expert model to obtain eighth intermediate data; and adding the sixth, seventh, and eighth intermediate data to obtain the second output data.

[0123] For example, in some specific implementations, the original BEiT3 does not have the ability to fuse multi-parameter MRI information. Therefore, multimodal fusion is performed by the last 12-k Transformer blocks through a soft MOME (i.e., MOME). Soft Completed. Here, the embedded features are concatenated as input to the adapted base model, denoted as X = [X...]. DCE,1 ,X DWI,1 ,X T2,1 The Transformer block can be represented as: Z l =X l +MSA*((LN*(X l )))(6) X l+1 =Z l +FFN*(LN*(Z l ))+MOME Soft (LN*(Z l ))(7)

[0124] MOME Soft The expression can be further refined to: MOME Soft (X)=LN(LP(SMoE(GELU(LP(X)))))(8)

[0125] Where X represents the input of the soft expert model; MOME Soft This represents the soft expert model; MOME Soft(X) represents the output of the soft expert model; SMoE is a soft expert hybrid (Soft MoE). Here, the linear projection layer is also used to reduce and expand the feature dimension, while SMoE is used to learn and integrate multimodal information.

[0126] It should also be noted that, in some preferred embodiments, without loss of generality, let X be the input of SMoE, where m is the number of labels in the input sequence data and d is the feature dimension. Specifically, soft MoE first obtains a set of slots through a linear combination of all input labels. The process can be further broken down as follows:

[0127] in, It is a learnable linear projection, where · represents the dot product, and D i,j Let represent the normalized weight assigned to the j-th slot by the i-th input label, where i represents the index of the label in the input sequence (i = 1, 2, ..., m), i′ represents the index of the label in the i′-th input sequence, j represents the index of the slot (j = 1, 2, ..., n·p), m represents the number of labels in the input sequence (i.e., the sequence length), and D represents the weight assigned to the j-th slot by D. i,j The constructed weight distribution matrix T represents the matrix transpose. Through the above process, n·p slots are generated, and each p slot is processed by an expert function, ultimately resulting in multiple experts. Let f represent the expert function; the subsequent process of soft MoE is to execute the expert function on the slots and map the slots back to the markers:

[0128] in, The expert function f is a linear projection function, C i,j This represents the normalized contribution weight of the j-th slot to the i-th input label, where j′ represents the index of the j′-th slot, and C represents the weight of C. i,j Constructed reverse weight distribution matrix Multiple expert functions are trained to learn different fusion features from long sequences of input generated by multi-parameter MRI. Subsequently, the combination of experts is used as an ensemble learning strategy to improve the fusion results.

[0129] Specifically, in this embodiment of the invention, the Soft Expert Hybrid (SMoE) aggregates multimodal information into a set of "slots" through learnable linear combination, which are then processed by different expert functions and finally mapped back to the token sequence. This mechanism can dynamically and selectively integrate the contributions of different modalities, simulating the decision-making process of multi-expert consultation, thereby achieving better and more intelligent multimodal information fusion, which is the key to improving the final classification performance.

[0130] Finally, after 12 Transformer blocks, the CLS markers are extracted, normalized, and then input into the linear classification layer. For inference of missing modalities, this invention only needs to remove the sparse experts corresponding to the missing sequences.

[0131] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0132] As shown in Figure 3, the technical solution of this embodiment of the invention can be specifically implemented as follows:

[0133] I. Dataset Characteristics:

[0134] This study involved 5,220 multiparametric breast MRI examinations from 5,205 patients. These data were collected from three institutions over a ten-year period, designated as Dataset 1 (DS1), Dataset 2 (DS2), and Dataset 3 (DS3). DS1 contains 1,824 examinations performed between November 2012 and July 2017. DS2 contains 735 examinations collected between December 2018 and March 2022, and DS3 contains 2,661 examinations obtained between November 2015 and October 2022. For malignancy classification, DS1 was randomly divided into a training set (n = 1,167) and a validation set (n = 150) for model development, an internal test set 1 (n = 307) for evaluation, and an internal test set 2 (n = 200) for comparison with radiologists. There was no overlap in patients between the different sets. DS2 and DS3 were used for external testing. In addition, triple-negative breast cancer (TNBC) subtyping and neoadjuvant chemotherapy (NACT) response prediction were performed using 1,005 and 358 participants in DS1, respectively, and their performance was reported through five-fold cross-validation.

[0135] The explanations of each section in Figure 3 are as follows: a. Data collection from three different hospitals, covering populations in northern, southeastern, and southwestern China. b. Generation of multiparameter breast MRI; this embodiment primarily uses T2-weighted MRI, diffusion-weighted MRI, and DCE-MRI. c. MOME first takes the multiparameter MRI as input. Then, based on the pre-trained base model parameters, it utilizes sparse modality expert mixture and soft modality expert mixture for single-modality feature extraction and multimodal information integration. d. MOME can be used for malignant tumor screening, molecular subtyping, and NACT (neoadjuvant chemotherapy) response prediction, providing non-invasive personalized management for breast cancer patients. DCE = Dynamic Contrast Enhancement; DWI = Diffusion-Weighted Imaging.

[0136] II. MRI Acquisition:

[0137] For DS1, magnetic resonance imaging (MRI) scans were performed on a 1.5T system (Magnetom Espree Pink; Siemens, Munich, Germany) using an 8-channel breast coil. The patient was positioned prone with both breasts naturally placed within the coil. Imaging included routine scans: axial T1-weighted 3D non-fat-suppressed sequences (TR / TE: 8.7 / 4.7 ms, matrix: 896 × 896, slice thickness: 1 mm), T2-weighted fat-suppressed sequences (TR / TE: 2900 / 60 ms, matrix: 640 × 640, slice thickness: 4 mm), and diffusion-weighted imaging (DWI) with b-values ​​of 400, 800, and 1000 s / mm. 2 (TR / TE: 6200 / 104ms, matrix: 236x120, slice thickness: 4mm). Dynamic contrast-enhanced MRI was performed using a 3D fat-suppressed VIBE sequence at six time points before and after bolus injection (0.1 mmol / kg gadopentetate dimeglumine, Magnevist, Bayer, Berlin, Germany), with an injection rate of 2 mL / s, followed by flushing with 20 mL of normal saline. The examination lasted 7 minutes, and the imaging parameters were TR / TE: 4.53 / 1.66ms, matrix: 384×384, slice thickness: 1.0 mm. Images from each time phase were automatically subtracted.

[0138] For DS2, breast MRI was performed using a 1.5T Magnetom Avanto or a 3.0T Magnetom Skyra MRI scanner (Siemens Healthineers, Erlangen, Germany) equipped with a dedicated breast coil. Patients underwent the examination in a prone position. The scans included the following sequences: axial T1-weighted non-fat-suppressed images with acquisition parameters of TR / TE 559 / 12 ms (1.5T) and 6 / 2.5 ms (3.0T), matrix 448×448, slice thickness 4 mm (1.5T) and 1.6 mm (3.0T); and axial T2-weighted images with acquisition parameters of TR / TE 4500 / 102 ms (1.5T) and 4740 / 107 ms (3.0T), matrix 512×512 (1.5T) and 448×448 (3.0T), slice thickness 4 mm. Diffusion-weighted images were acquired using a single-shot excitation planar echo imaging pulse sequence with the following parameters: TR / TE 6400 / 97ms (1.5T) and 5700 / 59ms (3.0T), matrix 192×192 (1.5T) and 340×170 (3.0T), slice thickness 4mm, and b-values ​​of 50 / 500 / 1000s / mm. 2 (1.5T) and 50 / 400 / 800s / mm 2(3.0T); DCE-MRI was performed during intravenous injection of 15 ml Gd-DTPA (0.1 mmol / kg), lasting 6 minutes and 41 seconds at a rate of 2.5 ml / s. The sequence included one axial image before contrast agent injection and five axial scans after contrast agent injection, spaced 30 seconds apart. DCE parameters were: TR / TE 5.2 / 2.4 ms (1.5T) and 4.7 / 1.7 ms (3.0T), matrix 384×384 (1.5T) and 448×448 (3.0T), slice thickness 1.1 mm (1.5T) and 1.6 mm (3.0T). Automatic subtraction was performed on images at each time phase.

[0139] For DS3, breast MRI was performed on a 1.5T system (Magnetom Avanto; Siemens, Germany) equipped with a 4-channel breast phased array surface coil. Patients underwent the examination in a prone position. The scanning procedure was as follows: axial T1WI fast small-angle excitation 3D, flash 3D sequence (TR 8.6ms, TE 4.7ms, slice thickness 1mm); fat-suppressed axial T2WI fast inversion recovery (TIRM) sequence (TR 5600ms, TE 56ms, slice thickness 4mm); axial diffusion-weighted imaging using single-shot echo-planar imaging (SS-EPI) sequence (TR 4900ms, TE 84ms, FOV 340mm, slice thickness 4mm, diffusion sensitivity b value selected as 0s / mm). 2 and 800s / mm 2 Axial dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) scan: First, a first-phase axial fat-suppressed T1WI (i.e., masked) scan was performed, followed by contrast agent injection, and then five consecutive phases of dynamic enhancement sequences, each lasting 60 seconds. 20 ml of Gd-DTPA-BMA (OmniScan, GE Healthcare, Ireland) was injected at a rate of 2.0 ml / s, followed by flushing with 20 ml of normal saline. DCE-MRI parameters were: TR 4.43 ms, TE 1.5 ms, matrix 352×324, slice thickness 1.7 mm. Automatic subtraction was performed on each phase of the image.

[0140] In some preferred embodiments, to improve data processing efficiency before model training or application, MRI data preprocessing can be performed: a three-dimensional breast region mask can be generated from T1-weighted fat-suppressed MRI images to remove irrelevant information and reduce the computational cost of MOME inference. Specifically, a two-dimensional binary breast mask is obtained for each MR image slice by extracting boundaries and applying morphological processing methods. Then, all two-dimensional masks are stacked to create a three-dimensional mask, which is smoothed using a 3D Gaussian filter (γ=20). The obtained 3D mask is used to crop the MRI scan and cover up air and chest areas. The main purpose of the breast mask is to reduce the dimensionality of the input data.

[0141] For DS1, this invention provides a pre-contrast T1WI and DCE-MRI at 6 time points. The invention uses DCE-MRI subtraction as model input, which enhances the tumor region. The invention first resizes a 3D mask to the same shape as the DCE-MRI images (384×384×128 in this invention's dataset). All DCE-MRI subtraction sequences are then cropped using the 3D mask (resulting sizes are typically smaller than 384×256×128 with 6 channels), and then normalized, linearly scaling the voxels to zero mean and unit variance. During training, all cropped MRI scans are again padded to the same size of 384×384×128. For T2 images, the size is first resized to 384×384×32, then the breast region is cropped, followed by normalization (linear scaling to zero mean and unit variance). For DWI, sequences with the highest b-values ​​(1000 or 800) are used, and normalization is performed only as preprocessing. During training and testing, DCE-MRI, T2WI and DWI were filled to 384×256×128, 384×256×48 and 256×128×32, respectively.

[0142] For DS2 and DS3, the procedure is similar to that of DS1, except that their DCE-MRI has 5 phases and uses first-order B-spline interpolation and grid constant mode interpolation to 6 phases.

[0143] All patients in DS1, DS2, and DS3 had malignant or benign labels confirmed by histopathological examination. In DS1, 365 patients with histopathologically confirmed breast cancer received NACT. One cycle of NACT lasted 21 days. After the second cycle, clinicians assessed each patient's response to and tolerability to NACT. All patients underwent MRI scans before treatment and at least two follow-up studies. All patients underwent definitive surgery after the final cycle of treatment. For confirmed breast cancer, molecular subtypes were determined according to the Chinese Anti-Cancer Association guidelines and analyzed by pathologists from the immunohistochemical results in the histopathological reports. Molecular subtypes were defined using estrogen receptor (ER), progesterone receptor (PR), HER2 status, and the Ki-67 index. Estrogen receptor and PR positivity were defined as staining greater than 1%. HER-2 positivity was defined as an IHC score of 3+ or fluorescence in situ hybridization amplification with a score of 2+ or higher. TNBC was determined by ER negativity, PR negativity, and HER2 negativity.

[0144] III. MOME:

[0145] MOME provides a unified and easily extensible architecture for multimodal data integration (such as multiparametric breast MRI). Input data is first embedded as features using different tokenizers, concatenated, and then passed to a Transformer architecture adapted from a base model.

[0146] Each MRI sequence uses a structurally identical tokenizer module to embed the input as tokens, i.e., a series of embedded features, denoted as X. DCE X DWI and X T2 The tokenizer consists of three 3D convolutional layers (stride = 2) and ends with a max-pooling layer (stride = 2). Each convolutional layer is followed by an instance normalization layer and a ReLU layer. The tokenizer downsamples the width, height, and number of slices of the MRI input by a factor of 1 / 16, generating a feature map with 768 feature dimensions. The resulting feature map is then flattened in the width, height, and slice dimensions to form the sequence input. For example, the original shape is... The DCE-MRI will be processed by the tokenizer into And flattened into A CLS token Attached to X DCE The data is then processed and used for final classification. Finally, a learnable one-dimensional positional embedding is added to the input token. The multimodal features from different sequences are then concatenated and fed into the subsequent Transformer structure.

[0147] MOME's Transformer architecture is adapted from BEiT3. BEiT3 was originally a visual language foundation model with 276 million parameters, pre-trained using 21 million image-text pairs, 14 million images, and 160GB of documents. This model contains twelve structurally identical Transformer blocks for feature encoding, the process of which can be described as follows: Z l =X l +MSA((LN(X l )))(1) X l+1 =Z l +FFN(LN(Z l ))(2)

[0148] Among them, X l It is the input of the l-th block, MSA represents multi-head self-attention mechanism, FFN represents feedforward network with two linear projection layers, LN represents layer normalization, X l+1 It outputs and inputs into the (l+1)th Transformer block.

[0149] Based on the BEiT3 model, this invention further introduces a Modal Expert Hybrid (MOME) to achieve multimodal learning and fusion. Specifically, this invention fixes the pre-training parameters of BEiT3 and injects simple trainable modules. This invention adds a sparse Modal Expert Hybrid to the Transformer block, setting the first k layers to learn from different modalities (i.e., different MRI sequences). The process can be represented as follows:

[0150] Where * indicates that the module parameters are loaded from the pre-trained BEiT3 and fixed during training, and i is the index. Specifically, It comes from set {X} DCE,1 ,X DWI,1 ,X T2,1 The i-th feature of} It is the i-th sparse MOME. Sparse A feedforward network structure employing two layers of linear projection followed by a normalized layer is used: MOME Sparse (X)=LN(LP(GELU(LP(X))))(5)

[0151] Here, LP represents a linear projection layer, and GELU represents a Gaussian error linear unit. In this way, each sparse MOME will learn to encode a specific type of input sequence, while most Transformer block parameters are fixed and shared across modalities.

[0152] The original BEiT3 does not have the ability to fuse multi-parameter MRI information. Therefore, multimodal fusion is performed by the last 12-k Transformer blocks through a soft MOME (i.e., MOME). Soft Completed. Here, the embedded features are concatenated as input to the adapted base model, denoted as X = [X...]. DCE,1 ,X DWI,1 ,X T2,1 The Transformer block can be represented as: Z l =X l +MSA*((LN*(X l )))(6) X l+1 =Z l +FFN*(LN*(Z l ))+MOME Soft (LN*(Z l ))(7)

[0153] MOME Soft The expression can be further refined to: MOME Soft (X)=LN(LP(SMoE(GELU(LP(X)))))(8)

[0154] SMoE is a hybrid of soft and soft expert (Soft MoE). Here, the linear projection layer is used to reduce and expand the feature dimension, while SMoE is used to learn and integrate multimodal information. Without loss of generality, let X be the input of SMoE, where m is the number of labels in the input sequence data and d is the feature dimension. Specifically, Soft MoE first obtains a set of slots through a linear combination of all input labels. The process can be further broken down as follows:

[0155] in, It is a learnable linear projection. Through the above process, n·p slots are generated, and each p slot is processed by an expert function, ultimately resulting in multiple experts. Let f represent the expert function. The subsequent process of soft MoE is to execute the expert function on the slot and map the slot back to the label:

[0156] in, The expert function f is a linear projection function. Multiple expert functions are trained to learn different fusion features from long sequences of input generated by multi-parameter MRI. Subsequently, the combination of experts is used as an ensemble learning strategy to improve the fusion results.

[0157] After passing through 12 Transformer blocks, CLS tags are extracted, normalized, and then input into the linear classification layer. For inference of missing modalities, this invention only needs to remove the sparse experts corresponding to the missing sequences.

[0158] Specifically, the first nine Transformer blocks of the MOME are implemented using a sparse MOME, while the last three Transformer blocks are implemented using a soft MOME. The soft MOME uses 128 experts, with each expert handling one input slot. The Adam optimizer is used, with an initial learning rate of 1×10⁻⁶. -4 The cosine annealing strategy was used to gradually reduce the temperature to 1×10⁻⁶. -6 .

[0159] For the malignant tumor classification task, each model was trained for 100 epochs with a batch size of 1, using the standard softmax function as the supervision signal. Results for this task were obtained through test-time augmentation, except when compared with other models. For the triple-negative breast cancer (TNBC) subtype classification and neoadjuvant chemotherapy (NACT) response prediction tasks, model parameters were initialized based on those trained on the malignant tumor classification task and trained for 200 epochs. NACT response prediction results were obtained through test-time augmentation. Both tasks used a weighted softmax function, with weights determined by the ratio of positive to negative samples. All implementations were based on PyTorch and used an NVIDIA GeForce RTX 3090 GPU.

[0160] During training, data augmentation methods such as random padding and random single-axis or dual-axis flipping can be used to improve data diversity. During inference, MOME generates patient-level predictions within the range (0,1), using the AUROC-optimal checkpoint on the validation set. During testing, nine different padding versions are generated from the input 3D MRI, combined with six different flipping scenarios, resulting in a total of 54 augmented versions. The final result is the average model score of these 54 augmented versions. A score greater than 0.5 is considered a positive case (e.g., breast cancer patients), otherwise a negative case (e.g., benign lesions).

[0161] To assess the trade-off between correctly downgraded cases and the true positive rate, this invention first adjusts the operating point on the DS1 internal test set, and then reports the results on DS2 using the resulting threshold.

[0162] For model interpretation, ensemble gradients were used to find salient regions on MRI images. The 3D visualization results of the ensemble gradients are shown in Figure 4 using MeVisLab. Shapley values ​​were calculated by inferring from seven combinations of three modalities. Specifically, Figure 4 corresponds to the superimposed 3D visualization of DCE subtraction (blocks marked a, b, g, h in Figure 4) with red saliency calculated by integral gradients, magnified axial views of DCE subtraction, DWI, and T2WI (blocks marked c, d, i, j in Figure 4), local Shapley values ​​(blocks marked e, f, k, l in Figure 4), and global Shapley values ​​of the DS1 internal test set (block marked m in Figure 4) and DS2 (block marked n in Figure 4). Specifically, Figure 4 shows four typical cases from the DS1 internal test set: a patient with a BI-RADS 5 malignant lesion (blocks marked a, c, e in Figure 4), a patient with a BI-RADS 4 benign lesion (blocks marked b, d, f in Figure 4), a patient with a BI-RADS 5 malignant lesion (blocks marked g, i, k in Figure 4), and a patient with a BI-RADS 4 benign lesion (blocks marked h, j, l in Figure 4). BI-RADS = Breast Imaging Reporting and Data System; DCE = Dynamic Contrast Enhancement; DWI = Diffusion-Weighted Imaging; T2WI = T2-Weighted Imaging. Source data are provided as source data files.

[0163] MOME is interpretable in highlighting lesions and analyzing the contribution of each modality. Local interpretation for each case can be analyzed using integral gradients and Shapley values ​​calculated for individual cases. Integral gradients show that MOME correctly focuses on breast lesions when diagnosing malignant (blocks a and g in Figure 4) or benign (blocks b and h and 5h in Figure 4), consistent with DS1 and DS2. Shapley values ​​reveal the contribution of each modality to the final prediction (blocks e, f, k, l in Figure 4). It can be observed that DCE and DWI play a more significant role in identifying malignant patients, while DCE and T2WI show a greater contribution in differentiating benign patients. A more comprehensive global interpretation for all cases can be derived from global Shapley values ​​(blocks m and n in Figure 4). It can be observed that DCE made the highest global contribution in identifying malignant tumors, while DWI and T2WI mainly contributed to the diagnosis of malignant and benign patients, respectively, and this decision rule was consistent in DS1 and DS2.

[0164] In some preferred embodiments, to investigate the impact of different modules, embodiments of the invention have also developed different variants of MOME by removing each individual component (the ablation portion of the method) and comparing their performance on the DS1 internal test set 1. As can be seen from Table 1, removing all modal experts results in a decrease in performance for all metrics (row 1). On the AUROC and AUPRC metrics, this variant shows a decrease of 1.0% and 1.1% respectively compared to MOME. The invention then replaces the soft modal experts with an MLP adapter (row 2), resulting in a 2.3% decrease in AUROC and a 3.0% decrease in AUPRC compared to MOME. Except for sensitivity, all other metrics show a decrease compared to the complete method. These findings indicate that each module contributes to improving the final results.

[0165] Table 1

[0166] Table 1 above shows the AUROC and AUPRC performance of the MOME ablation studies. Results are reported as mean values ​​and 95% confidence intervals based on 1000 bootstrapping samples. No MOME (w / o MOME): All modality experts removed; No soft MOME (w / o MOME) Soft Remove the last three layers of the soft expert hybrid.

[0167] In summary, this invention presents a large-scale modal expert mixture model (MOME) based on FM (Base Model). It inherits the long-range modeling capabilities of Transformer-based FM for multi-parameter information fusion and allows for flexible inference through expert mixture (MoE) design. Developed and extensively evaluated based on the largest multi-parameter breast MRI dataset collected from various hospitals, this model achieves malignancy identification performance comparable to that of National Health Commission (NHC) certified radiologists. The model is universally applicable to data from different regions, demonstrating its clinical value in reducing unnecessary biopsies in BI-RADS 4 patients. Furthermore, MOME is capable of triple-negative breast cancer subtyping and neoadjuvant chemotherapy response prediction. With these capabilities, this invention demonstrates the clinical value of MOME in non-invasive, personalized management of breast cancer patients.

[0168] As shown in Figure 5, this embodiment of the invention also provides a breast cancer auxiliary decision-making device 900 based on multi-parameter MRI, which can implement the above-described method. This device may include:

[0169] The first module 910 is used to acquire the MRI dataset; wherein, each sample data in the MRI dataset includes multi-parameter MRI and label data of the same breast sample, and the multi-parameter MRI includes MRI sequences of different modalities;

[0170] The second module 920 is used to extract features from the multi-parameter MRI of the sample data to obtain multimodal features;

[0171] The third module 930 is used to optimize and train the preset model using multimodal features and label data to obtain the target model;

[0172] The preset model includes a preset number of Transformer blocks. In the preset model, a sparse modal expert mixture is added to the first number of Transformer blocks, and a soft expert model is added to the second number of Transformer blocks. The sum of the first number and the second number is the preset number. The sparse modal expert mixtures in different Transformer blocks are configured to process MRI sequences from different modalities within the same Transformer block.

[0173] Module 4, 940, is used for breast cancer auxiliary decision-making using the target model.

[0174] In some embodiments, the apparatus further includes a fifth module for performing the following operations:

[0175] Obtain a general multimodal base model; wherein the general multimodal base model includes the preset number of Transformer blocks;

[0176] The sparse modal expert hybrid is added to the first number of Transformer blocks before the general multimodal base model, and the soft expert model is added to the second number of Transformer blocks after the general multimodal base model to construct the initial model;

[0177] The sparse modal expert hybrid contains multiple parallel sparse expert networks configured for different modalities within each Transformer block;

[0178] The pre-trained parameters of the general multimodal base model are loaded into the corresponding shared layer in the initial model, thereby initializing the parameters of the sparse modal expert hybrid and the soft expert model, and configuring them to obtain the preset model;

[0179] In the preset model, the first number of Transformer blocks are configured to encode the MRI sequences of different modalities within a single block using different sparse expert networks.

[0180] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0181] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0182] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0183] As shown in Figure 6, Figure 6 illustrates the hardware structure of an electronic device 1000 according to another embodiment. The electronic device 1000 includes:

[0184] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0185] The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001.

[0186] Input / output interface 1003 is used to implement information input and output;

[0187] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0188] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0189] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0190] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0191] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0192] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0193] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0194] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0195] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0196] The present invention provides a method, device, electronic device, storage medium, and program product for breast cancer assisted decision-making based on multi-parameter MRI. This method acquires an MRI dataset; each sample in the MRI dataset includes multi-parameter MRI and label data of the same breast sample, and the multi-parameter MRI includes MRI sequences of different modalities. Features are extracted from the multi-parameter MRI of the sample data to obtain multi-modal features; a preset model is optimized and trained using the multi-modal features and label data to obtain a target model; the preset model includes a preset number of Transformer blocks, wherein: a first number of Transformer blocks contain sparse modal expert mixtures, and a second number of Transformer blocks contain soft expert models, the sum of the first and second numbers being the preset number; the sparse modal expert mixtures in different Transformer blocks are configured to process MRI sequences from different modalities within the same Transformer block; and the target model is used for breast cancer assisted decision-making. This invention introduces a sparse modal expert hybrid in the preceding Transformer block, enabling the model to learn specific feature representations for different MRI modalities. Furthermore, it introduces a soft expert model in the subsequent Transformer block, achieving deep fusion and interactive modeling of multimodal information. This hierarchical and structured expert hybrid design overcomes the limitations of traditional methods in multimodal fusion, effectively improving the model's accuracy and robustness in the differential diagnosis of breast cancer, and laying a technical foundation for achieving non-invasive and personalized management.

[0197] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0198] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0199] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0201] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A breast cancer decision support method based on multi-parameter MRI, characterized in that, The method includes the following steps: Obtain an MRI dataset; wherein each sample in the MRI dataset includes multi-parameter MRI and label data of the same breast sample, and the multi-parameter MRI includes MRI sequences of different modalities; Feature extraction is performed on the multi-parameter MRI of the sample data to obtain multimodal features; The preset model is optimized and trained using the multimodal features and the label data to obtain the target model; The preset model includes a preset number of Transformer blocks. In the preset model, a sparse modal expert hybrid is added to a first number of Transformer blocks, and a soft expert model is added to a second number of Transformer blocks. The sum of the first number and the second number is the preset number. The sparse modal expert hybrids in different Transformer blocks are configured to process the MRI sequences from different modalities within the same Transformer block. The target model was used for breast cancer auxiliary decision-making.

2. The method according to claim 1, characterized in that, The multiparameter MRI includes T2-weighted imaging, diffusion-weighted imaging, and dynamic contrast-enhanced imaging. The feature extraction of the multiparameter MRI data from the sample data to obtain multimodal features includes the following steps: The MRI sequences are embedded using a tokenizer to obtain the embedding features corresponding to each modality of the MRI sequences. The embedded features are flattened in dimensions, and then a preset CLS token and position embedding are added to obtain the input features; The input features corresponding to the MRI sequences of different modalities are concatenated to form the multimodal features.

3. The method according to claim 2, characterized in that, The tokenizer includes multiple convolutional layers and a max-pooling layer. The embedding operation of the MRI sequence using the tokenizer includes the following steps: The height, width, and number of slices of the MRI sequence are downsampled by a preset factor through multiple convolutional layers and max pooling layers; The preset multiplier is determined based on the product of the stride of all the convolutional layers and the stride of the max pooling layer.

4. The method according to claim 1, characterized in that, The method further includes the following steps: Obtain a general multimodal base model; wherein the general multimodal base model includes the preset number of Transformer blocks; The sparse modal expert hybrid is added to the first number of Transformer blocks before the general multimodal base model, and the soft expert model is added to the second number of Transformer blocks after the general multimodal base model to construct the initial model; The sparse modal expert hybrid contains multiple parallel sparse expert networks configured for different modalities within each Transformer block; The pre-trained parameters of the general multimodal base model are loaded into the corresponding shared layer in the initial model, thereby initializing the parameters of the sparse modal expert hybrid and the soft expert model, and configuring them to obtain the preset model; In the preset model, the first number of Transformer blocks are configured to encode the MRI sequences of different modalities within a single block using different sparse expert networks.

5. The method according to claim 1, characterized in that, The multimodal features are appended with a CLS token. The step of optimizing and training a preset model using the multimodal features and the label data to obtain a target model includes the following steps: The multimodal features are input into the preset model, and after being processed by a preset number of Transformer blocks, the CLS token corresponding to the CLS token is extracted. The CLS tags are subjected to layer normalization and linear classification to obtain the training inference results; Based on the training inference results and the labeled data, a loss function is constructed, and then the preset model is tuned and optimized to obtain the target model.

6. The method according to claim 1, characterized in that, The method of using the target model for breast cancer auxiliary decision-making includes the following steps: Acquire the true multiparametric MRI images to be processed; The feature extraction is performed on the real multiparameter MRI to obtain real multimodal features as the first input data; wherein, the real multimodal features are appended with a CLS token; The first Transformer block in the target model is used as the first processing block; The first input data is input into the first processing block, and the first output data is obtained by processing based on the sparse modal expert hybrid. The next Transformer block is used as the first processing block, the first output data is used as the first input data, and the step of inputting the first input data into the first processing block is returned to be executed until the number of the first processing blocks reaches the first number. The next Transformer block is used as the second processing block, and the first output data is used as the second input data; The second input data is input into the second processing block, and the second output data is obtained based on the soft expert model. The next Transformer block is used as the second processing block, the second output data is used as the second input data, and the step of inputting the second input data into the second processing block is returned to be executed until the number of the second processing blocks reaches the second number. Wherein, the sum of the first quantity and the second quantity is the preset quantity; Extract the CLS token corresponding to the CLS token from the second output data output from the last Transformer block; The CLS markers are subjected to hierarchical normalization and linear classification to obtain prediction inference results, which serve as the basis for breast cancer auxiliary decision-making.

7. The method according to claim 6, characterized in that, The first processing block includes a multi-head self-attention mechanism, a feedforward network, layer normalization, and the sparse modality expert hybrid. The sparse modality expert hybrid contains multiple parallel sparse expert networks configured for different modalities within each Transformer block. The step of inputting the first input data into the first processing block and processing it based on the sparse modality expert hybrid to obtain the first output data includes the following steps: The first input data is processed sequentially through the layer normalization and the multi-head self-attention mechanism to obtain the first intermediate data; Add the first intermediate data to the first input data to obtain the second intermediate data; The second intermediate data is passed sequentially through the layer normalization and the feedforward network to obtain the third intermediate data; The second intermediate data is subjected to layer normalization, and the results of the layer normalization are divided into multiple modal subsequences according to the modal source; Each of the modal subsequences is input into the corresponding sparse expert network for feature mapping to obtain multiple expert output sequences. The multiple expert output sequences are concatenated along the sequence dimension to obtain the fourth intermediate data; The expression for the sparse modal expert hybrid is: MOME Sparse (X)=LN(LP(FREEZE(LP(X)))), In the formula, X represents the input of the sparse modal expert hybrid; MOME Sparse MOME represents a sparse modal expert hybrid. Sparse (X) represents the output of the sparse modal expert mixture; LN represents layer normalization; LP represents the linear projection layer; GELU represents the Gaussian error linear unit; The second intermediate data, the third intermediate data, and the fourth intermediate data are added together to obtain the first output data.

8. The method according to claim 6, characterized in that, The second processing block includes a multi-head self-attention mechanism, a feedforward network, layer normalization, and the soft expert model. The step of inputting the second input data into the second processing block and processing it based on the soft expert model to obtain the second output data includes the following steps: The second input data is processed sequentially through the layer normalization and the multi-head self-attention mechanism to obtain the fifth intermediate data; The fifth intermediate data is added to the second input data to obtain the sixth intermediate data; The sixth intermediate data is passed sequentially through the layer normalization and the feedforward network to obtain the seventh intermediate data; The sixth intermediate data is then processed sequentially through the layer normalization and the soft expert model to obtain the eighth intermediate data; The expression for the soft expert model is as follows: MY Soft (X)=LN(LP(SMoE(GELU(LP(X)))))), In the formula, X represents the input of the soft expert model; MOME Soft This represents the soft expert model; MOME Soft (X) represents the output of the soft expert model; LN represents layer normalization; LP represents the linear projection layer; SMoE represents the soft expert mixture; GELU represents the Gaussian error linear unit. The sixth intermediate data, the seventh intermediate data, and the eighth intermediate data are added together to obtain the second output data.

9. A breast cancer auxiliary decision-making device based on multi-parameter MRI, characterized in that, The device includes: The first module is used to acquire an MRI dataset; wherein, each sample data in the MRI dataset includes multi-parameter MRI and label data of the same breast sample, and the multi-parameter MRI includes MRI sequences of different modalities; The second module is used to extract features from the multi-parameter MRI of the sample data to obtain multimodal features; The third module is used to optimize and train the preset model using the multimodal features and the label data to obtain the target model; The preset model includes a preset number of Transformer blocks. In the preset model, a sparse modal expert hybrid is added to a first number of Transformer blocks, and a soft expert model is added to a second number of Transformer blocks. The sum of the first number and the second number is the preset number. The sparse modal expert hybrids in different Transformer blocks are configured to process the MRI sequences from different modalities within the same Transformer block. The fourth module is used to assist in breast cancer decision-making using the target model.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images