Weakly supervised histopathological image segmentation method based on fuzzy-enhanced transformer

Abstract

The present invention belongs to the technical field of intelligent medical image processing. Provided is a weakly supervised histopathological image segmentation method based on a fuzzy-enhanced Transformer. The technical problem of it being difficult to put into practical application due to the high cost of acquiring medical image data labels is solved. The technical solution comprises: first, reading an image from a histopathological data set, and performing data preprocessing; then, inputting the image into a Transformer, and acquiring attention matrices and a final layer output; next, using a fuzzy-enhanced attention module to generate a fuzzy-enhanced attention map for guiding a pooling operation to train the Transformer model; and finally, extracting a fuzzy-enhanced attention map from the model, and performing post-processing to generate a pathological image segmentation result. The beneficial effect of the present invention is that accurate histopathological image segmentation is realized by using readily available image-level labels, thereby prompting the practical application of deep learning in the field of medical diagnosis.

Description

A Weakly Supervised Histopathological Image Segmentation Method Based on Fuzzy Enhancement Transformer Technical Field

[0001] This invention relates to the field of intelligent medical image processing technology, and in particular to a weakly supervised tissue pathology image segmentation method based on fuzzy enhancement Transformer. Background Technology

[0002] Segmentation of histopathological images plays a crucial role in medical image analysis, particularly in the diagnosis of tumors, inflammation, and other pathological changes. Accurate segmentation of different tissue regions allows physicians to better identify the type, extent, and invasiveness of tumors, providing vital information for clinical treatment and prognostic assessment. Especially in the study of the tumor microenvironment, segmenting different tissue types such as tumor epithelium, tumor-infiltrating lymphocytes, and tumor-associated stroma is significant for revealing tumor biological behavior, predicting treatment response, and assessing patient prognosis. With the continuous development of deep learning technology, image segmentation methods based on convolutional neural networks (CNNs) and Transformers have become mainstream. Deep learning models can automatically learn complex feature representations from data, greatly improving the efficiency and accuracy of medical image segmentation, thereby assisting physicians in clinical decision-making.

[0003] For example, Van et al. proposed HookNet, a multi-branch segmentation framework based on convolutional neural networks, in their paper "HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images." This framework generates fine-grained segmentation maps for histopathological images by combining contextual information and high-resolution details. However, the collection of medical image annotation data is often expensive and time-consuming, especially for high-resolution histopathological images such as whole-slide images (WSI), whose gigapixel resolution makes manual annotation extremely difficult and labor-intensive. Furthermore, the high heterogeneity of tumor regions leads to diverse morphologies and tissue structures, posing even greater challenges to image segmentation tasks. There is an urgent need for a new segmentation method that can utilize simple and readily available image-level labels for pixel-level histopathological image segmentation, thereby reducing the reliance on large amounts of high-quality annotation data. Simultaneously, by incorporating theoretical methods such as fuzzy logic, this method can effectively address the problem of blurred tissue category boundaries in histopathological images, improving the accuracy and robustness of weakly supervised segmentation. This is significant for building reliable and low-cost medical decision support systems. Summary of the Invention

[0004] The purpose of this invention is to provide a weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer, which solves the problems of existing histopathological image segmentation methods such as dependence on high-quality data annotation and blurred tissue category boundaries, and effectively improves the accuracy of weakly supervised segmentation.

[0005] To achieve the above-mentioned objectives, this invention is implemented through the following measures: a weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer, comprising the following steps:

[0006] S1: Read images from the histopathology dataset Perform data preprocessing operations, and then cut the image into 196 non-overlapping image blocks of size 16×16;

[0007] S2: Construct input tokens from preprocessed histopathological images. The input is fed into the Transformer encoder layer to obtain the attention matrix and extract the affinity matrix. Final layer image block token output and category token output Where D is the dimension of the token;

[0008] S3: Output the image block token The fuzzy enhanced attention module constructed from the reshaped input extracts fuzzy features from the image block token output based on fuzzy rules. The fuzzy features are then concatenated with the image block token output and fused through a convolution operation to generate a fuzzy enhanced attention map.

[0009] S4: The generated blurred enhanced attention map is processed and used as weights to guide the pooling operation, generating image block token prediction scores. The category token output is dimensionality reduced to generate category token prediction scores. The two scores are compared with the image's true labels to calculate the multi-label soft edge loss and train the Transformer model.

[0010] S5: Extract the blurred enhanced attention map from the trained model, apply the ReLU activation function and maximum normalization operation, then perform post-processing using the affinity matrix, and finally generate the segmentation result after the argmax operation, where argmax is the operation to obtain the index of the position of the maximum value.

[0011] Furthermore, the specific steps of step S2 are as follows:

[0012] Step S2.1: Input tensor T input The input Transformer encoder layer undergoes L iterations, during which a query matrix is ​​generated through projection transformation. Key matrix Sum matrix Extracting the attention matrix and final layer token output

[0013]

[0014] T output =AV (2)

[0015] Where d k Let be the dimension of the key matrix, and softmax(·) be the activation function. The aggregated attention matrix is ​​obtained by aggregating the attention matrices of layers L.

[0016] Step S2.2: Apply the aggregated attention matrix A all The average attention matrix is ​​obtained by averaging across the layers.

[0017]

[0018] in Let i be the aggregate attention matrix;

[0019] Step S2.3: From the average attention matrix A mean Extracting the affinity matrix Output T from the final level token output Extract the final layer image block token output and category token output

[0020] Furthermore, the specific steps of step S3 are as follows:

[0021] Step S3.1: Output the extracted image block tokens Shape reshaping Represented as D units of size The feature maps are then analyzed, and fuzzy membership functions are used to characterize the association between elements in the feature maps and fuzzy concepts, effectively capturing the uncertainty and fuzziness in the features. K Gaussian membership functions are assigned to each feature map, and the formula for calculating the Gaussian membership function is as follows:

[0022]

[0023] Where c d,k and σ d,k Tables 1 and 2 show the center and width of the k-th Gaussian membership function on the d-th feature map, where d = 1, ..., D, k = 1, ..., K, exp(·) is the exponential function, and T is the width of the Gaussian membership function. dLet μ represent the d-th feature map. d,k (T d ) represents the fuzzy membership matrix obtained by assigning the k-th Gaussian membership function on the d-th feature map. The final fuzzification result is calculated using formula (4).

[0024] Step S3.2: Sample the fuzzy Gaussian kernel to synthesize fuzzy features with semantic information. The sampling formula is as follows:

[0025]

[0026] Where T' d,k This represents the feature maps obtained through sampling, where the pixel values ​​follow a mean of c. d,k The standard deviation is Gaussian distribution;

[0027] Step S3.3: Different membership values ​​represent the membership degrees of different locations in the image to a specific fuzzy set. Three sets of fuzzy rules are established to extract fuzzy weights, which are then multiplied with the sampled feature maps to extract fuzzy features. The first set of fuzzy logic is used to extract highly matched semantic fuzzy features. The maximum membership value of each feature point on the feature map is selected as the output weight. To obtain highly matching semantically ambiguous features Implementing fuzzy logic using scaling softmax operation:

[0028]

[0029] Where τ is the scaling temperature parameter controlling the smoothness, and the second set of fuzzy logic is used to extract semantically fuzzy features with low matching, selecting the minimum membership value of each feature point on the feature map as the output weight. To obtain semantically ambiguous features with low matching Implementing fuzzy logic using scaling softmax operation:

[0030]

[0031] The third set of fuzzy logic is used to extract general semantic fuzzy features, selecting the average membership value of each feature point on the feature map as the output weight. To obtain general semantic fuzzy features Implementing fuzzy logic using scaling softmax operation:

[0032]

[0033] in For blurring results The result of taking the mean along the second dimension;

[0034] Step S3.4: After concatenating the three extracted fuzzy features with the original features, perform a convolution operation to fuse them:

[0035]

[0036] Concat(·) is a concatenation operation, and Conv(·) is a 3×3 convolution operation. The concatenated feature channels are adjusted to C, where C is the number of classes in the dataset. Finally, a blurred and enhanced attention map is generated.

[0037] Furthermore, the specific steps of step S4 are as follows:

[0038] Step S4.1: The blurred attention map contains positional information to guide the pooling operation. First, softmax normalization is performed on the channel dimension to generate pooling weights α. c,x,y :

[0039]

[0040] in This represents the feature point in the c-th channel of the blurred enhanced attention map, where c = 1, ..., C.

[0041] Step S4.2: With α c,x,y For weighted pairs A fuzzy Weighted pooling is applied to increase the weight of the correct category at the corresponding position, thereby guiding the attention map to expand within a reasonable range and generating image patch token prediction scores.

[0042]

[0043] Step S4.3: Output the category token obtained in step S2.3 Category token prediction scores are generated through linear layer dimensionality reduction.

[0044] y class =Linear(T) class (15)

[0045] Where Linear(·) represents a linear layer;

[0046] Step S4.4: Image block token prediction score With category token prediction score Respectively with real labels

[0047] Calculate the multi-label soft edge loss:

[0048]

[0049] Where σ(·) represents the sigmoid activation function, and finally L patch and L class The sum is used as the final loss function, and the model parameters are optimized by minimizing the loss function through backpropagation.

[0050] Furthermore, the specific steps of step S5 are as follows:

[0051] Step S5.1: After the model training is complete, extract the blur-enhanced attention map after feeding the image back into the model. The fuzzy enhanced attention map corresponding to each category is then processed using the ReLU activation function and maximum normalization:

[0052]

[0053] in The fuzzy enhanced attention map corresponding to category c, where max(·) represents the maximum value operation;

[0054] Step S5.2: Use the affinity matrix extracted in step S2.3 Fine-grained post-processing of the blurred enhanced attention map:

[0055]

[0056] in R represents matrix multiplication operations. N×C (·)and These represent reshaping the matrix into N×C and respectively. The shape;

[0057] Step S5.3: Finally, place A final Upsample to the input image size and apply the argmax operation to generate the final segmentation result, pred:

[0058]

[0059] Upsample(·) represents the upsampling operation. This indicates that the maximum index is taken in the category dimension, and the segmentation result is visualized.

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0061] 1. This invention proposes a weakly supervised learning method based on image-level labels, which can effectively alleviate the high cost and high labor intensity problems in the medical image annotation process. Unlike traditional methods that require a large amount of pixel-level labeled data, this invention can achieve high-precision pixel-level segmentation through image-level labels, greatly reducing the need for data annotation and lowering annotation costs and time consumption.

[0062] 2. This invention proposes a fuzzy-enhanced class-aware attention map generation method. Multiple Gaussian membership functions and fuzzy rules are set on the output feature map to extract fuzzy high-matching semantic features, fuzzy low-matching semantic features, and fuzzy general semantic features. The fuzzy-enhanced attention map generated by fusing these fuzzy features can provide accurate object position information. At the same time, the fuzzy-enhanced attention map can guide the pooling operation to reasonably expand the attention region.

[0063] 3. This invention combines the fuzzy enhanced attention map in the trained model with the self-attention map in the sub-attention module of the Transformer model to generate the final attention map result as the segmentation result of the histopathological image. The segmentation result can accurately identify key tissue types such as tumor regions, tumor-infiltrating lymphocytes, and tumor-related stroma, providing strong support for the segmentation of histopathological images. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used to explain the invention but do not constitute a limitation thereof.

[0065] Figure 1 is an overall framework diagram of the weakly supervised tissue pathology image segmentation method based on fuzzy enhancement Transformer of the present invention;

[0066] Figure 2 is a flowchart of the weakly supervised tissue pathology image segmentation method based on fuzzy enhancement Transformer of the present invention;

[0067] Figure 3 is a flowchart of the fuzzy enhanced attention module of the present invention;

[0068] Figure 4 shows an example of the weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0070] Example 1

[0071] Referring to Figures 1 to 4, this embodiment provides a technical solution: a weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer. Taking a histopathological image dataset as an example, the process from loading data to obtaining segmentation results includes the following steps:

[0072] S1: Read images from the histopathology dataset Perform data preprocessing operations, and then cut the image into 196 non-overlapping image blocks of size 16×16;

[0073]

[0074] S2: Construct input tokens from preprocessed histopathological images. The input is fed into the Transformer encoder layer to obtain the attention matrix and extract the affinity matrix. Final layer image block token output and category token output Where D is the dimension of the token;

[0075] S3: Output the image block token The fuzzy enhanced attention module constructed from the reshaped input extracts fuzzy features from the image block token output based on fuzzy rules. The fuzzy features are then concatenated with the image block token output and fused through a convolution operation to generate a fuzzy enhanced attention map.

[0076] S4: The generated blurred enhanced attention map is processed and used as weights to guide the pooling operation, generating image block token prediction scores. The category token output is dimensionality reduced to generate category token prediction scores. The two scores are compared with the image's true labels to calculate the multi-label soft edge loss and train the Transformer model.

[0077] S5: Extract the blurred enhanced attention map from the trained model, apply the ReLU activation function and maximum normalization operation, then perform post-processing using the affinity matrix, and finally generate the segmentation result after the argmax operation, where argmax is the operation to obtain the index of the position of the maximum value.

[0078] Furthermore, the specific steps of step S2 are as follows:

[0079] Step S2.1: Input tensor T input The input Transformer encoder layer undergoes L iterations, during which a query matrix is ​​generated through projection transformation. Key matrix Sum matrix Extracting the attention matrix and final layer token output

[0080]

[0081] T output =AV (3)

[0082] Where d k Let be the dimension of the key matrix, and softmax(·) be the activation function. The aggregated attention matrix is ​​obtained by aggregating the attention matrices of layers L.

[0083] Step S2.2: Apply the aggregated attention matrix A all The average attention matrix is ​​obtained by averaging across the layers.

[0084]

[0085] in Let i be the aggregate attention matrix;

[0086] Step S2.3: From the average attention matrix A mean Extracting the affinity matrix Output T from the final level token output Extract the final layer image block token output and category token output A affinity T patch and T class As shown in formulas (5), (6) and (7):

[0087]

[0088] T class =(0.36540.0239...-1.3109) 1×384 (7)

[0089] Furthermore, the specific steps of step S3 are as follows:

[0090] Step S3.1: Output the extracted image block tokens Shape reshaping Represented as D units of size The feature maps are then analyzed, and fuzzy membership functions are used to characterize the association between elements in the feature maps and fuzzy concepts, effectively capturing the uncertainty and fuzziness in the features. K Gaussian membership functions are assigned to each feature map, and the formula for calculating the Gaussian membership function is as follows:

[0091]

[0092] Where c d,k and σ d,k Tables 1 and 2 show the center and width of the k-th Gaussian membership function on the d-th feature map, where d = 1, ..., D, k = 1, ..., K, exp(·) is the exponential function, and T is the width of the Gaussian membership function. d Let μ represent the d-th feature map. d,k (T d ) represents the fuzzy membership matrix obtained by assigning the k-th Gaussian membership function on the d-th feature map. The final fuzzification result is calculated using formula (8). Where the membership matrix μ 0,0 As shown in formula (9):

[0093]

[0094] Step S3.2: Sample the fuzzy Gaussian kernel to synthesize fuzzy features with semantic information. The sampling formula is as follows:

[0095]

[0096] Where T' d,k This represents the feature maps obtained through sampling, where the pixel values ​​follow a mean of c. d,k The standard deviation is The Gaussian distribution, where the sampled feature map T' 0,0 As shown in formula (11):

[0097]

[0098] Step S3.3: Different membership values ​​represent the membership degrees of different locations in the image to a specific fuzzy set. Three sets of fuzzy rules are established to extract fuzzy weights, which are then multiplied with the sampled feature maps to extract fuzzy features. The first set of fuzzy logic is used to extract highly matched semantic fuzzy features. The maximum membership value of each feature point on the feature map is selected as the output weight. To obtain highly matching semantically ambiguous features Implementing fuzzy logic using scaling softmax operation:

[0099]

[0100]

[0101] Where τ is the scaling temperature parameter that controls the smoothness, and the high-match semantic fuzzy features. As shown in formula (14):

[0102]

[0103] The second set of fuzzy logic is used to extract semantically fuzzy features with low matching, selecting the minimum membership value of each feature point on the feature map as the output weight. To obtain semantically ambiguous features with low matching Implementing fuzzy logic using scaling softmax operation:

[0104]

[0105] low-match semantic ambiguity features As shown in formula (17):

[0106]

[0107] The third set of fuzzy logic is used to extract general semantic fuzzy features, selecting the average membership value of each feature point on the feature map as the output weight. To obtain general semantic fuzzy features Implementing fuzzy logic using scaling softmax operation:

[0108]

[0109] in For blurring results The result of averaging along the second dimension shows low-match semantic ambiguity. As shown in formula (20):

[0110]

[0111] Step S3.4: After concatenating the three extracted fuzzy features with the original features, perform a convolution operation to fuse them:

[0112]

[0113] Concat(·) is a concatenation operation, and Conv(·) is a 3×3 convolution operation. The concatenated feature channels are adjusted to C, where C is the number of classes in the dataset. Finally, a blurred and enhanced attention map is generated. As shown in formula (22):

[0114]

[0115] Furthermore, the specific steps of step S4 are as follows:

[0116] Step S4.1: The blurred attention map contains positional information to guide the pooling operation. First, softmax normalization is performed on the channel dimension to generate pooling weights α. c,x,y:

[0117]

[0118] in This represents the feature point in the c-th channel of the blurred enhanced attention map, where c = 1, ..., C. α c,x,y As shown in formula (24):

[0119]

[0120] Step S4.2: With α c,x,y For weighted pairs A fuzzy Weighted pooling is applied to increase the weight of the correct category at the corresponding position, thereby guiding the attention map to expand within a reasonable range and generating image patch token prediction scores.

[0121]

[0122] y patch As shown in formula (26):

[0123] y patch = (7.7413 - 5.4955 - 8.54437.2036) 1×4 (26)

[0124] Step S4.3: Output the category token obtained in step S2.3 Category token prediction scores are generated through linear layer dimensionality reduction.

[0125] y class =Linear(T) class (27)

[0126] Where Linear(·) represents a linear layer, y class As shown in formula (28):

[0127] y class = (8.2471 -4.5638 -8.5215 6.5667) 1×4 (28)

[0128] Step S4.4: Image block token prediction score With category token prediction score Respectively with real labels Calculate the multi-label soft edge loss:

[0129]

[0130] Where σ(·) represents the sigmoid activation function, and finally L patch and L class The sum is used as the final loss function, and the model parameters are optimized by minimizing the loss function through backpropagation.

[0131] Furthermore, the specific steps of step S5 are as follows:

[0132] Step S5.1: After the model training is complete, extract the blur-enhanced attention map after feeding the image back into the model. The fuzzy enhanced attention map corresponding to each category is then processed using the ReLU activation function and maximum normalization:

[0133]

[0134] in The fuzzy enhanced attention map corresponding to category c, where max(·) represents the maximum value operation. As shown in formula (32):

[0135]

[0136] Step S5.2: Use the affinity matrix extracted in step S2.3 Fine-grained post-processing of the blurred enhanced attention map:

[0137]

[0138] in R represents matrix multiplication operations. N×C (·)and These represent reshaping the matrix into N×C and respectively. The shape, A final As shown in formula (34):

[0139]

[0140] Step S5.3: Finally, place A final Upsample to the input image size and apply the argmax operation to generate the final segmentation result, pred:

[0141]

[0142] Where Upsample(·) represents the upsampling operation. This indicates that the maximum index is taken in the category dimension, and the segmentation result is visualized. The segmentation result pred is shown in formula (36):

[0143]

[0144] All images in the dataset were fed into the model to obtain segmentation results. The performance of the segmentation results was evaluated using the real segmentation labels. The accuracy was 84.51% and the average intersection-over-union ratio was 75.65%, which is significantly better than traditional weakly supervised segmentation methods.

[0145]

[0146] Example 2

[0147] Referring to Figures 1 to 4, this embodiment provides a technical solution: a weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer. Taking a histopathological image dataset as an example, the process from loading data to obtaining segmentation results includes the following steps:

[0148] S1: Read images from the histopathology dataset Perform data preprocessing operations, and then cut the image into 196 non-overlapping image blocks of size 16×16;

[0149]

[0150] S2: Construct input tokens from preprocessed histopathological images. The input is fed into the Transformer encoder layer to obtain the attention matrix and extract the affinity matrix. Final layer image block token output and category token output Where D is the dimension of the token;

[0151] S3: Output the image block token The fuzzy enhanced attention module constructed from the reshaped input extracts fuzzy features from the image block token output based on fuzzy rules. The fuzzy features are then concatenated with the image block token output and fused through a convolution operation to generate a fuzzy enhanced attention map.

[0152] S4: The generated blurred enhanced attention map is processed and used as weights to guide the pooling operation, generating image block token prediction scores. The category token output is dimensionality reduced to generate category token prediction scores. The two scores are compared with the image's true labels to calculate the multi-label soft edge loss and train the Transformer model.

[0153] S5: Extract the blurred enhanced attention map from the trained model, apply the ReLU activation function and maximum normalization operation, then perform post-processing using the affinity matrix, and finally generate the segmentation result after the argmax operation, where argmax is the operation to obtain the index of the position of the maximum value.

[0154] Furthermore, the specific steps of step S2 are as follows:

[0155] Step S2.1: Input tensor T input The input Transformer encoder layer undergoes L iterations, during which a query matrix is ​​generated through projection transformation. Key matrix Sum matrix Extracting the attention matrix and final layer token output

[0156]

[0157] T output =AV (3)

[0158] Where d k Let be the dimension of the key matrix, and softmax(·) be the activation function. The aggregated attention matrix is ​​obtained by aggregating the attention matrices of layers L.

[0159] Step S2.2: Apply the aggregated attention matrix A all The average attention matrix is ​​obtained by averaging across the layers.

[0160]

[0161] in Let i be the aggregate attention matrix;

[0162] Step S2.3: From the average attention matrix A mean Extracting the affinity matrix Output T from the final level token output Extract the final layer image block token output and category token output A affinity T patch and T class As shown in formulas (5), (6) and (7):

[0163]

[0164] T class =(-10.1635-0.3858...18.8144) 1×384 (7)

[0165] Furthermore, the specific steps of step S3 are as follows:

[0166] Step S3.1: Output the extracted image block tokens Shape reshaping Represented as D units of size The feature maps are then analyzed, and fuzzy membership functions are used to characterize the association between elements in the feature maps and fuzzy concepts, effectively capturing the uncertainty and fuzziness in the features. K Gaussian membership functions are assigned to each feature map, and the formula for calculating the Gaussian membership function is as follows:

[0167]

[0168] Where c d,k and σ d,k Tables 1 and 2 show the center and width of the k-th Gaussian membership function on the d-th feature map, where d = 1, ..., D, k = 1, ..., K, exp(·) is the exponential function, and T is the width of the Gaussian membership function. d Let μ represent the d-th feature map. d,k (T d ) represents the fuzzy membership matrix obtained by assigning the k-th Gaussian membership function on the d-th feature map. The final fuzzification result is calculated using formula (8). Where the membership matrix μ 0,0 As shown in formula (9):

[0169]

[0170] Step S3.2: Sample the fuzzy Gaussian kernel to synthesize fuzzy features with semantic information. The sampling formula is as follows:

[0171]

[0172] Where T' d,k This represents the feature maps obtained through sampling, where the pixel values ​​follow a mean of c. d,k The standard deviation is The Gaussian distribution, where the sampled feature map T' 0,0 As shown in formula (11):

[0173]

[0174] Step S3.3: Different membership values ​​represent the membership degrees of different locations in the image to a specific fuzzy set. Three sets of fuzzy rules are established to extract fuzzy weights, which are then multiplied with the sampled feature maps to extract fuzzy features. The first set of fuzzy logic is used to extract highly matched semantic fuzzy features. The maximum membership value of each feature point on the feature map is selected as the output weight. To obtain highly matching semantically ambiguous features Implementing fuzzy logic using scaling softmax operation:

[0175]

[0176] Where τ is the scaling temperature parameter that controls the smoothness, and the high-match semantic fuzzy features. As shown in formula (14):

[0177]

[0178] The second set of fuzzy logic is used to extract semantically fuzzy features with low matching, selecting the minimum membership value of each feature point on the feature map as the output weight. To obtain semantically ambiguous features with low matching Implementing fuzzy logic using scaling softmax operation:

[0179]

[0180] low-match semantic ambiguity features As shown in formula (17):

[0181]

[0182] The third set of fuzzy logic is used to extract general semantic fuzzy features, selecting the average membership value of each feature point on the feature map as the output weight. To obtain general semantic fuzzy features Implementing fuzzy logic using scaling softmax operation:

[0183]

[0184] in For blurring results The result of averaging along the second dimension shows low-match semantic ambiguity. As shown in formula (20):

[0185]

[0186] Step S3.4: After concatenating the three extracted fuzzy features with the original features, perform a convolution operation to fuse them:

[0187]

[0188] Concat(·) is a concatenation operation, and Conv(·) is a 3×3 convolution operation. The concatenated feature channels are adjusted to C, where C is the number of classes in the dataset. Finally, a blurred and enhanced attention map is generated. As shown in formula (22):

[0189]

[0190] Furthermore, the specific steps of step S4 are as follows:

[0191] Step S4.1: The blurred attention map contains positional information to guide the pooling operation. First, softmax normalization is performed on the channel dimension to generate pooling weights α. c,x,y :

[0192]

[0193] in This represents the feature point in the c-th channel of the blurred enhanced attention map, where c = 1, ..., C. α c,x,y As shown in formula (24):

[0194]

[0195] Step S4.2: With α c,x,y For weighted pairs A fuzzy Weighted pooling is applied to increase the weight of the correct category at the corresponding position, thereby guiding the attention map to expand within a reasonable range and generating image patch token prediction scores.

[0196]

[0197] y patch As shown in formula (26):

[0198] y patch = (7.7469 - 5.9008 - 8.12457.8962) 1×4 (26)

[0199] Step S4.3: Output the category token obtained in step S2.3 Category token prediction scores are generated through linear layer dimensionality reduction.

[0200] y class =Linear(T) class (27)

[0201] Where Linear(·) represents a linear layer, y class As shown in formula (28):

[0202] y class = (6.8711 - 7.5792 - 8.94048.5667) 1×4 (28)

[0203] Step S4.4: Image block token prediction score With category token prediction score Respectively with real labels Calculate the multi-label soft edge loss:

[0204]

[0205] Where σ(·) represents the sigmoid activation function, and finally L patch and L class The sum is used as the final loss function, and the model parameters are optimized by minimizing the loss function through backpropagation.

[0206] Furthermore, the specific steps of step S5 are as follows:

[0207] Step S5.1: After the model training is complete, extract the blur-enhanced attention map after feeding the image back into the model. The fuzzy enhanced attention map corresponding to each category is then processed using the ReLU activation function and maximum normalization:

[0208]

[0209] in The fuzzy enhanced attention map corresponding to category c, where max(·) represents the maximum value operation. As shown in formula (32):

[0210]

[0211] Step S5.2: Use the affinity matrix extracted in step S2.3 Fine-grained post-processing of the blurred enhanced attention map:

[0212]

[0213] in R represents matrix multiplication operations. N×C (·)and These represent reshaping the matrix into N×C and respectively. The shape, A final As shown in formula (34):

[0214]

[0215] Step S5.3: Finally, place A final Upsample to the input image size and apply the argmax operation to generate the final segmentation result, pred:

[0216]

[0217] Where Upsample(·) represents the upsampling operation. This indicates that the maximum index is taken in the category dimension, and the segmentation result is visualized. The segmentation result pred is shown in formula (36):

[0218]

[0219] All images in the dataset were fed into the model to obtain segmentation results. The performance of the segmentation results was evaluated using the real segmentation labels. The accuracy was 83.68% and the average intersection-over-union ratio was 67.68%, which is significantly better than traditional weakly supervised segmentation methods.

[0220]

[0221] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A weakly supervised histopathological image segmentation method based on fuzzy augmented Transformer, characterized in that, The method comprises the following steps: S1 : Read images from histopathology dataset Wherein H and W are the height and width of the image respectively, data preprocessing operations are performed, and then the picture is cut into N non-overlapping image blocks with a size of P*P, wherein N is the number of image blocks, and P is the side length of the image block; S2: constructing the pre-processed histopathological image into input tokens transformer encoder layer, and extracting an affinity matrix from the attention matrix of the transformer encoder layer Final layer image tile token output and category token output Wherein D is the dimension of the token; S3: output the image block token The reshaped input constructs a fuzzy enhancement attention module, extracts fuzzy features in the image block token output based on fuzzy rules, and fuses the fuzzy features and the image block token output through convolution operation to generate a fuzzy enhancement attention map; S4: The generated fuzzy enhancement attention map is processed as a weight to guide the pooling operation, and the image block token prediction score is generated. The class token output is processed to generate a class token prediction score. The two scores are respectively calculated with the image real label to calculate the multi-label soft edge loss to train the Transformer model; S5: Extract the fuzzy enhancement attention map in the trained model, pass it through the ReLU activation function and the max normalization operation, then use the affinity matrix for post-processing, and generate the segmentation result after upsampling and argmax operation, wherein argmax is the operation of obtaining the position index of the maximum value.

2. The weakly supervised histopathological image segmentation method based on blur enhanced Transformer according to claim 1, wherein, The specific steps of the step S2 are as follows: Step S2.1 : inputting a tensor T input The input Transformer encoder layer undergoes L rounds of iteration, and in the iteration process, the query matrix is generated through projection transformation key matrix and value matrix extracting an attention matrix and final layer token output T output = AV(2) where d k is the dimension of the key matrix, softmax(·) is an activation function, and the aggregated attention matrix is obtained by aggregating the attention matrices of the L layers Step S2.2: Aggregating the attention matrices A all Averaging over the dimension of the layers yields an average attention matrix wherein The i-th aggregated attention matrix is Step S2.3: Extracting affinity matrix from average attention matrix A mean ​ from the final layer token output T output extracting the final layer image block token output and category token output 3. The weakly supervised histopathological image segmentation method based on blur enhanced Transformer according to claim 1, wherein, The specific steps of the step S3 are as follows: Step S3.1: output the extracted image block tokens Shape remodelling is denoted by D number of size The feature map is characterized, and then the fuzzy membership function is used to characterize the relevance of the elements in the feature map and the fuzzy concept, effectively capturing the uncertainty and fuzziness in the features, and K Gaussian membership functions are assigned to each feature map, and the calculation formula of the Gaussian membership function is as follows: where c d,k and σ d,k Table respectively the center and width of the kth Gaussian membership function on the dth feature map, d = 1,..., D, k = 1,..., K, exp(·) is the exponential function, T d represents the dth feature map, μ d,k (T d ) represents the fuzzy membership matrix obtained by assigning the kth Gaussian membership function on the dth feature map, and the final fuzzy result is calculated using formula (4) Step S3.2: Sampling the blurred Gaussian kernel for synthesizing the blurred feature with semantic information, the formula of sampling is as follows: wherein representing the feature maps obtained by sampling, the pixel values on these feature maps being subject to a mean c d,k , a standard deviation Gaussian distribution Step S3.3: Different membership values represent the membership of different positions in the image to a certain fuzzy set. Three sets of fuzzy rules are established to extract fuzzy weights, which are multiplied by the feature maps obtained by sampling to extract fuzzy features. The first set of fuzzy logic is used to extract high-matching semantic fuzzy features, and the maximum membership value of each feature point on the feature map is selected as the output weight To obtain high matching semantic fuzzy features Fuzzy logic is implemented using scaled softmax operations: where μ d,k represents the fuzzy membership matrix obtained by assigning the kth Gaussian membership function on the dth feature map, τ is a scaling temperature parameter that controls the smoothness, the second set of fuzzy logic is used to extract the low matching semantic fuzzy features, and the minimum membership value of each feature point on the feature map is selected as the output weight to obtain low matching semantic ambiguous features Fuzzy logic is implemented using scaled softmax operations: The third fuzzy logic is used to extract the general semantic fuzzy features, and the medium membership value of each feature point on the feature map is selected as the output weight to obtain general semantic fuzzy features Fuzzy logic is implemented using scaled softmax operations: wherein To fuzz the result The result of taking the mean in the second dimension Step S3.4: The extracted three blur features are spliced with the original features and fused using a convolution operation: where Concat(·) is a concatenation operation, Conv(·) is a 3x3 convolution operation, the concatenated feature channels are adjusted to C, C is the number of classes of the dataset, and a fuzzy enhanced attention map is finally generated 4. The weakly supervised histopathological image segmentation method based on blur enhanced Transformer according to claim 1, wherein, The specific steps of the step S4 are as follows: Step S4.1: The attention map contains location information, which is used to guide the pooling operation. First, softmax normalization is performed in the channel dimension to generate the pooling weight α c,x,y : wherein representing a feature point on the c-th channel in the blur enhanced attention map, c = 1,..., C, Step S4.2: Perform weighted pooling on A c,x,y with α fuzzy as the weight, enhance the weight of the correct class in the corresponding position, so as to guide the attention map to expand within a reasonable range, and generate the image block token prediction score Step S4.3: output the category token acquired in Step S2.3 Generating class token prediction scores by linear layer dimensionality reduction y class = Linear(T class ) (15) Wherein Linear(·) represents a linear layer; Step S4.4: Image block token prediction score Class token prediction scores respectively to the true label Computing multi-label soft margin loss: where σ(·) denotes a sigmoid activation function, and L patch and L class are added as the final loss function, which is minimized by backpropagation to optimize the model parameters.

5. The weakly supervised histopathological image segmentation method based on blur enhanced Transformer according to claim 1, wherein, The specific steps of the step S5 are as follows: Step S5.1: After the model training is completed, the image is put into the model again to extract the blur enhancement attention map and the corresponding fuzzy enhancement attention map of each category is activated by a ReLU activation function and maximum normalization operation: wherein The fuzzy enhancement attention map corresponding to the class c, max(·) represents the maximum value operation; Step S5.2: Use the affinity matrix extracted in step S2.3 performing fine processing on the blur-enhanced attention map: wherein denotes a matrix multiplication operation, R N×C (·) and respectively denote reshaping a matrix to N x C and The shape of Step S5.3: Finally, A final The up-sampling to the input image size and applying argmax operation generates the final segmentation result pred: Upsample (·) denotes an up-sampling operation, where The maximum value index is taken in the class dimension, and the segmentation result is visualized and displayed.

Images