General pathology task processing method and related device

Abstract

A general pathology task processing method and a related device. The method comprises: on the basis of data samples of a tissue dataset, using a preset student model to perform processing to obtain a student global feature token and student local feature tokens; on the basis of the data samples, using an expert model corresponding to a preset pathology task to perform processing to obtain an expert global feature token and expert local feature tokens; constructing a class token alignment loss on the basis of the student global feature token and the expert global feature token, and constructing a patch token alignment loss on the basis of the student local feature tokens and the expert local feature tokens; summing the class token alignment loss and the patch token alignment loss to obtain a total loss; on the basis of the total loss, using a teacher-student network strategy to perform configuration to obtain a teacher network; and using the teacher network to process the pathology task. In the method, the student model directly learns expert features, which can improve the training efficiency and performance, avoid separate repetitive training for each type of task, and be widely applied to the technical field of imaging data processing.

Description

A general method and related equipment for pathology task processing Technical Field

[0001] This invention relates to the field of imaging data processing technology, and in particular to a general pathology task processing method and related equipment. Background Technology

[0002] Pathology plays a crucial role in modern medicine, providing essential information for disease diagnosis, treatment, and prognosis. In recent years, with the rapid development of digital pathology, especially whole-slice imaging technology, computational pathology has become an important research field, aiming to use computational methods to analyze digital pathological images to assist clinical decision-making. However, pathology tasks are diverse, encompassing various types such as slice-level classification, survival prediction, tissue classification, image retrieval, visual question answering, and report generation, and the data annotations required for each task are limited. This makes training a model from scratch for each independent task both time-consuming and impractical.

[0003] In recent years, self-supervised learning based on large-scale datasets has driven the development of foundational models in computational pathology. Existing research has proposed various foundational pathology models, such as UNI, Phikon, and CONCH, which perform well on specific task types. However, these models are typically evaluated on a limited number of task types, lack a comprehensive generalization assessment system, and no model has yet demonstrated superior performance across all types of clinical tasks. This task-specific performance indicates that existing foundational models are limited by their training data distribution and pre-training strategies, making it difficult to adapt to the diverse clinical needs in computational pathology. Summary of the Invention

[0004] The main objective of this invention is to provide a general pathology task processing method, apparatus, electronic device, storage medium, and program product, aiming to solve at least one problem of the prior art.

[0005] To achieve the above objectives, one aspect of the present invention proposes a general pathology task processing method, the method comprising:

[0006] Obtain the tissue dataset; the data samples in the tissue dataset include image patches corresponding to whole-slice images;

[0007] Based on the data samples, the global and local feature labels of students are obtained by using a pre-defined student model.

[0008] Based on the data samples, expert global feature labels and expert local feature labels are obtained by using the expert model corresponding to the preset pathology task.

[0009] A class label alignment loss is constructed based on student global feature labels and expert global feature labels, and an image patch label alignment loss is constructed based on student local feature labels and expert local feature labels;

[0010] The total loss is obtained by summing the class label alignment loss and the image patch label alignment loss.

[0011] Based on the total loss, the teacher network is configured using a teacher-student network strategy.

[0012] Using a faculty network to handle pathology tasks.

[0013] In some embodiments, obtaining an organizational dataset includes the following steps:

[0014] Collect the original dataset; the original dataset includes multiple full-slice images;

[0015] Based on the original dataset, image patches of a preset size containing tissue were extracted from each full-slice image using a preset image tool, and then the tissue dataset was obtained.

[0016] In some embodiments, a class label alignment loss is constructed based on student global feature labels and expert global feature labels, including the following steps:

[0017] A first similarity assessment is performed on the student global feature labels and the expert global feature labels to obtain the first cosine similarity between the student global feature labels and the expert global feature labels;

[0018] The class label alignment loss is constructed based on the complementary value of the first cosine similarity compared to 1.

[0019] In some embodiments, when a preset pathology task includes multiple sub-tasks, the expert global feature label includes the sub-expert global feature label obtained by the expert model processing for each sub-task. A class label alignment loss is constructed based on the student global feature label and the expert global feature label, including the following steps:

[0020] A second similarity evaluation is performed on the global feature labels of students and the global feature labels of each sub-expert to obtain the second cosine similarity between the global feature labels of students and the global feature labels of each sub-expert.

[0021] The subclass label alignment loss for each subtask is constructed based on the complementary value of the second cosine similarity compared to 1.

[0022] The class label alignment loss is obtained by performing a first weighted summation on the subclass label alignment loss corresponding to all subtasks.

[0023] In some embodiments, an image patch label alignment loss is constructed based on student local feature labels and expert local feature labels, including the following steps:

[0024] A third similarity assessment is performed on the local feature labels of students and experts to obtain the third cosine similarity between the local feature labels of students and experts.

[0025] A first distance evaluation is performed on the student local feature labels and the expert local feature labels to obtain the first smoothed L1 distance between the student local feature labels and the expert local feature labels;

[0026] The first image patch loss is constructed based on the complementary value of the third cosine similarity compared to 1, and the second image patch loss is constructed based on the complementary value of the first smooth L1 distance compared to 1.

[0027] The image block label alignment loss is obtained by performing a second weighted summation on the first image block loss and the second image block loss.

[0028] In some embodiments, when a preset pathology task includes multiple sub-tasks, the expert local feature labels include sub-expert local feature labels obtained by the expert model processing for each sub-task. The image patch label alignment loss is constructed based on the student local feature labels and the expert local feature labels, including the following steps:

[0029] A fourth similarity evaluation is performed on the local feature labels of the students and the local feature labels of each sub-expert to obtain the fourth cosine similarity between the local feature labels of the students and the local feature labels of each sub-expert.

[0030] A second distance evaluation is performed on the student's local feature label and the local feature label of each sub-expert to obtain the second smoothed L1 distance between the student's local feature label and the local feature label of each sub-expert.

[0031] The third image patch loss for each subtask is constructed based on the complementary value of the fourth cosine similarity compared to 1, and the fourth image patch loss for each subtask is constructed based on the complementary value of the second smooth L1 distance compared to 1.

[0032] The third image patch loss and the fourth image patch loss of each subtask are summed by a third weight to obtain the sub-image patch label alignment loss of that subtask.

[0033] The image patch label alignment loss is obtained by performing a fourth weighted summation on the sub-image patch label alignment loss corresponding to all sub-tasks.

[0034] In some embodiments, the teacher network is configured using a teacher-student network strategy based on the total loss, including the following steps:

[0035] Based on the total loss, the weight parameters of the student model are updated through backpropagation;

[0036] Based on the updated weight parameters of the student model, the weights of the preset teacher network are updated using an exponential moving average to obtain the target teacher network.

[0037] To achieve the above objectives, another aspect of the present invention provides a universal pathology task processing device, the device comprising:

[0038] The first module is used to obtain the tissue dataset; the data samples of the tissue dataset include image patches corresponding to full-slice images;

[0039] The second module is used to process data samples using a pre-defined student model to obtain global and local feature labels for students.

[0040] The third module is used to obtain expert global feature labels and expert local feature labels based on data samples and using the expert model corresponding to the preset pathology task.

[0041] The fourth module is used to construct class label alignment loss based on student global feature labels and expert global feature labels, and to construct image patch label alignment loss based on student local feature labels and expert local feature labels;

[0042] The fifth module is used to sum up the class label alignment loss and the image patch label alignment loss to obtain the total loss;

[0043] The sixth module is used to configure the teacher network based on the total loss and the teacher-student network strategy.

[0044] Module 7 is used to process pathology tasks using a teacher network.

[0045] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0046] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0047] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0048] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a general pathology task processing method, device, electronic device, storage medium, and program product. This solution obtains a tissue dataset; wherein, the data samples of the tissue dataset include image patches corresponding to whole-slice images; based on the data samples, a preset student model is used to process and obtain student global feature labels and student local feature labels; based on the data samples, a preset expert model corresponding to the pathology task is used to process and obtain expert global feature labels and expert local feature labels; a class label alignment loss is constructed based on the student global feature labels and expert global feature labels, and an image patch label alignment loss is constructed based on the student local feature labels and expert local feature labels; the class label alignment loss and the image patch label alignment loss are summed to obtain a total loss; based on the total loss, a teacher network is configured using a teacher-student network strategy; and the teacher network is used to process the pathology task. This invention integrates expert model knowledge by pre-setting expert models corresponding to pathology tasks, enabling the construction of a general pathology foundation model and avoiding repeated training for each task. Furthermore, this invention utilizes a dual alignment loss of class labels and image patch labels, allowing student models to directly learn expert features, effectively improving training efficiency and performance. Additionally, this invention employs a teacher-student network and an exponential moving average strategy to enhance model convergence stability and generalization ability, thereby supporting the subsequent expansion of new expert models. This invention provides a complete process from data preparation to task processing, reducing system integration and deployment complexity. Attached Figure Description

[0049] Figure 1 is a schematic diagram of an implementation environment for a general pathology task processing method provided in an embodiment of the present invention.

[0050] Figure 2 is a flowchart illustrating a general pathology task processing method provided in an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of an example of a GPFM overview provided in an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of a comprehensive comparison example of various basic models provided in the embodiments of the present invention in 72 tasks;

[0053] Figure 5 is a schematic diagram showing the performance comparison of various basic models provided in the embodiments of the present invention in the whole-slice image classification task;

[0054] Figure 6 is a schematic diagram showing the performance comparison of various basic models provided in the embodiments of the present invention in 15 survival analysis tasks;

[0055] Figure 7 is a schematic diagram showing the performance comparison of various basic models provided in the embodiments of the present invention in the organizational classification task;

[0056] Figure 8 is a schematic diagram of an example of an overview of pathological tissue retrieval, visual question answering, and report generation provided in an embodiment of the present invention;

[0057] Figure 9 is a schematic diagram of an example of evaluating the effectiveness of expert knowledge distillation provided in an embodiment of the present invention;

[0058] Figure 10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0060] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0061] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0062] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0063] Among related technologies, the task-specific performance of existing basic models is evident, but due to limitations in their training data distribution and pre-training strategies, they are difficult to adapt to the diverse clinical needs in computational pathology.

[0064] In view of this, embodiments of the present invention provide a general pathology task processing method and related equipment. This method involves acquiring a tissue dataset; wherein the data samples of the tissue dataset include image patches corresponding to whole-slice images; based on the data samples, a preset student model is used to process and obtain student global feature labels and student local feature labels; based on the data samples, a preset expert model corresponding to the pathology task is used to process and obtain expert global feature labels and expert local feature labels; a class label alignment loss is constructed based on the student global feature labels and expert global feature labels, and an image patch label alignment loss is constructed based on the student local feature labels and expert local feature labels; the class label alignment loss and the image patch label alignment loss are summed to obtain a total loss; based on the total loss, a teacher network is configured using a teacher-student network strategy; and the teacher network is used to process the pathology task. This invention integrates expert model knowledge by pre-setting expert models corresponding to pathology tasks, enabling the construction of a general pathology foundation model and avoiding repeated training for each task. Furthermore, this invention utilizes a dual alignment loss of class labels and image patch labels, allowing student models to directly learn expert features, effectively improving training efficiency and performance. Additionally, this invention employs a teacher-student network and an exponential moving average strategy to enhance model convergence stability and generalization ability, thereby supporting the subsequent expansion of new expert models. This invention provides a complete process from data preparation to task processing, reducing system integration and deployment complexity.

[0065] It is understood that the general pathology task processing method provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various types of terminals or servers. When the computer device in the embodiments is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet computer, laptop computer, or desktop computer, but it is not limited to these.

[0066] Figure 1 illustrates an implementation environment according to an embodiment of the present invention. Referring to Figure 1, this implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0067] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0068] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0069] Terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0070] For example, based on the implementation environment shown in FIG1, this embodiment of the invention provides a general pathology task processing method. The following description takes the application of this general pathology task processing method in server 101 as an example. It can be understood that this general pathology task processing method can also be applied in terminal 102.

[0071] Referring to Figure 2, which is an optional flowchart of a general pathology task processing method provided by an embodiment of the present invention, the executing entity of this general pathology task processing method can be any of the aforementioned computer devices (including servers or terminals). The method in Figure 2 may include, but is not limited to, steps S100 to S700.

[0072] Step S100: Obtain the organization dataset;

[0073] The data samples in the organization dataset include image patches corresponding to full-slice images;

[0074] It should be noted that in some embodiments, step S100 may include the following steps: acquiring an original dataset; wherein the original dataset includes multiple full-slice images; based on the original dataset, using a preset image tool to extract image blocks of preset specifications containing tissue from each full-slice image, and then organizing them to obtain a tissue dataset.

[0075] Specifically, the embodiments of the present invention enhance the model's adaptability to pathological images of different scales by uniformly extracting non-overlapping, fixed-size image blocks while preserving the original resolution, thereby improving the robustness and generalization of feature extraction.

[0076] For example, in some specific implementations, the OpenSlide and CLAM toolkits can be used to extract 512×512 non-overlapping image patches from a full-slice image at level 0 resolution, while preserving the original image resolution, and to construct a dataset containing the corresponding image patches.

[0077] Step S200: Based on the data samples, the student global feature labels and student local feature labels are obtained by processing the data using a preset student model;

[0078] For example, in some specific implementations, a student network (i.e., a student model) is used to extract encoded class ([CLS]) tags (i.e., student global feature tags) and [PATCH] tags (i.e., student local feature tags), specifically:

[0079] [CLS] tag: is a "global summary" or "overall representation" of the entire image patch, and is a single, high-dimensional feature vector.

[0080] [PATCH] markers: These are the "local features" of each small part after an image patch has been segmented; they are a set of feature vectors.

[0081] The following is a detailed breakdown using pathological images:

[0082] 1. [CLS] Classification Token:

[0083] It is an additional, learnable vector placed at the very beginning of the input sequence. How it works: When the model processes an image patch (e.g., a 512x512 pathological region), the [CLS] tag interacts fully with all other [PATCH] tags in the Transformer layers (through self-attention). In this process, it "browses" and "summarizes" all the local information of the entire image patch. After all the network layers, the output vector of this [CLS] tag encodes the global, overall semantic information about this image patch.

[0084] 2. [PATCH] tokens:

[0085] It is a series of feature vectors obtained after an image patch is segmented and linearly projected, with each vector representing a local region in the original image. How it works: A 512x512 image patch is typically further subdivided into smaller grids (e.g., 16x16 pixels), each small grid being a "patch". Each patch is flattened and mapped to a feature vector, which is a [PATCH] tag. Therefore, an image patch will correspond to tens to hundreds of [PATCH] tags (e.g., 512x512 divided into 16x16 patches will result in 1024 patches). Each [PATCH] tag encodes the visual features of its corresponding local region.

[0086] Step S300: Based on the data sample, the expert global feature label and expert local feature label are obtained by processing the data using the expert model corresponding to the preset pathology task.

[0087] For example, in some specific implementations, the [CLS] (i.e., expert global feature marker) and [PATCH] marker (expert local feature marker) are obtained using the expert models employed. Specifically, the principle logic for [CLS] and [PATCH] is similar to that of the aforementioned student model, and will not be repeated here.

[0088] Step S400: Construct class label alignment loss based on student global feature labels and expert global feature labels, and construct image patch label alignment loss based on student local feature labels and expert local feature labels;

[0089] It should be noted that, in some embodiments, constructing a class label alignment loss based on student global feature labels and expert global feature labels may include the following steps: performing a first similarity evaluation on student global feature labels and expert global feature labels to obtain a first cosine similarity between student global feature labels and expert global feature labels; and constructing a class label alignment loss based on the complementary value of the first cosine similarity compared to 1.

[0090] Specifically, in this embodiment of the invention, cosine similarity is used to measure the consistency of global features between the student model and the expert model, ensuring that the student model can effectively learn the global semantic representation of the expert model.

[0091] For example, in some specific implementations, taking the UNI model as an example, the cosine similarity between the [CLS] label (sc, i.e., student global feature label) output by the student model and the [CLS] label (ac, i.e. expert global feature label) output by the UNI model is calculated, and 1-cos(sc,ac) is used as the class label alignment loss.

[0092] It should be noted that when the preset pathology task includes multiple sub-tasks, and the expert global feature label includes the sub-expert global feature label obtained by the expert model processing for each sub-task, in some embodiments, the class label alignment loss is constructed based on the student global feature label and the expert global feature label, which may include the following steps: performing a second similarity evaluation on the student global feature label and each sub-expert global feature label to obtain the second cosine similarity between the student global feature label and each sub-expert global feature label; constructing the sub-class label alignment loss corresponding to each sub-task based on the complementary value of the second cosine similarity compared to 1; and performing a first weighted summation on the sub-class label alignment losses corresponding to all sub-tasks to obtain the class label alignment loss.

[0093] Specifically, in this embodiment of the invention, a multi-expert weighted fusion mechanism enables the student model to learn the advantages of multiple experts in different tasks simultaneously, thereby further enhancing the model's comprehensive generalization ability.

[0094] For example, in some specific implementations, the preset pathology task may include a classification task, a report generation task, and a visual question answering task. A model that performs well in classification (UNI), report generation (Phikon), and visual question answering (CONCH) can then be selected as an expert model. The subclass label alignment loss (based on cosine similarity) between the student's [CLS] label and the [CLS] labels of UNI, Phikon, and CONCH is calculated separately. Weight parameters for each task are set, and a weighted sum is performed to obtain the final class label alignment loss.

[0095] It should be noted that, in some embodiments, constructing the image patch label alignment loss based on student local feature labels and expert local feature labels may include the following steps: performing a third similarity evaluation on student local feature labels and expert local feature labels to obtain a third cosine similarity between student local feature labels and expert local feature labels; performing a first distance evaluation on student local feature labels and expert local feature labels to obtain a first smoothed L1 distance between student local feature labels and expert local feature labels; constructing a first image patch loss based on the complementary value of the third cosine similarity compared to 1, and constructing a second image patch loss based on the complementary value of the first smoothed L1 distance compared to 1; and performing a second weighted summation on the first image patch loss and the second image patch loss to obtain the image patch label alignment loss.

[0096] Specifically, this embodiment of the invention combines cosine similarity and smoothed L1 distance to align local features from both similarity and distance perspectives, thereby enhancing the model's learning ability on detailed features.

[0097] For example, in some specific implementations, taking the UNI expert model as an example, the cosine similarity (applying the cosine function) and the smooth L1 distance (applying the smooth L1 function) between the student's [PATCH] label (sp, i.e., the student's local feature label) and the UNI's [PATCH] label (ap, i.e., the expert's local feature label) are calculated, and then weighted and summed after being assigned weights η and θ respectively, to obtain the image patch label alignment loss: d ap =η*(1-cos(sp,ap))+θ*SmoothL1(sp,ap).

[0098] It should be noted that when the preset pathology task includes multiple sub-tasks, and the expert local feature labels include the sub-expert local feature labels obtained by the expert model processing for each sub-task, in some embodiments, the image patch label alignment loss is constructed based on the student local feature labels and the expert local feature labels, which may include the following steps: performing a fourth similarity evaluation on the student local feature labels and each sub-expert local feature label to obtain the fourth cosine similarity between the student local feature labels and each sub-expert local feature label; performing a second distance evaluation on the student local feature labels and each sub-expert local feature label to obtain the second smoothed L1 distance between the student local feature labels and each sub-expert local feature label; constructing a third image patch loss for each sub-task based on the complementary value of the fourth cosine similarity compared to 1, and constructing a fourth image patch loss for each sub-task based on the complementary value of the second smoothed L1 distance compared to 1; performing a third weighted summation on the third image patch loss and the fourth image patch loss for each sub-task to obtain the sub-image patch label alignment loss for that sub-task; and performing a fourth weighted summation on the sub-image patch label alignment losses for all sub-tasks to obtain the image patch label alignment loss.

[0099] Specifically, the embodiments of the present invention can support multi-expert local feature alignment, and through a weighted fusion mechanism, enable the model to absorb multi-source knowledge at the detail level, thereby improving the diversity and discriminative power of feature representation.

[0100] For example, in some specific implementations, the preset pathology task may include a classification task, a report generation task, and a visual question answering task. A model that performs well in classification (UNI), report generation (Phikon), and visual question answering (CONCH) can then be selected as the expert model. The sub-image patch label alignment loss (based on cosine similarity and smoothed L1 distance) between the student's [PATCH] label and the [PATCH] labels of UNI, Phikon, and CONCH is calculated separately. After weighting and summing the weight parameters for each task, the image patch label alignment loss for multiple experts is obtained.

[0101] Step S500: Sum the class label alignment loss and the image patch label alignment loss to obtain the total loss;

[0102] For example, in some specific implementations, the class tag alignment loss d can be... c Image patch label alignment loss d p The total loss d is obtained by directly adding them together.

[0103] Step S600: Based on the total loss, the teacher network is configured using a teacher-student network strategy.

[0104] It should be noted that in some embodiments, step S600 may include the following steps: updating the weight parameters of the student model through backpropagation based on the total loss; updating the weights of the preset teacher network through exponential moving average based on the updated weight parameters of the student model to obtain the target teacher network.

[0105] Specifically, the embodiments of the present invention use exponential moving average to update the teacher network, so that the model remains stable during training, which can avoid the impact of student model fluctuations on knowledge distillation effect and improve training convergence and final performance.

[0106] It should be understood that the student model and the teacher model use the same network structure, such as the ViT-L network. Theoretically, GPFM can use all common backbone network structures, such as ResNet50, the ViT series, etc., but this embodiment of the invention does not limit the specific network structure.

[0107] For example, in some specific implementations, in each training step, the student model weights are updated via backpropagation using the total loss, and then the teacher network weights are updated via EMA, ultimately resulting in a stable general pathology foundation model GPFM (i.e., the target teacher network).

[0108] Step S700: Process pathology tasks using the teacher network;

[0109] For example, in some specific implementations, once the model training is complete, the whole slice image WSI can be split first, and then features can be extracted from the split image blocks. Based on the extracted features, various downstream tasks can be performed, such as benign and malignant classification, lesion detection, and report generation, depending on the preset pathological task requirements.

[0110] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0111] First, it's important to note that pre-trained foundational models on large-scale datasets are revolutionizing the field of computational pathology (CPath). The generalization ability of these foundational models is crucial for their success across various downstream clinical tasks. However, existing foundational models have only been evaluated on a limited number and types of tasks, and their generalization ability and overall performance remain unclear. To fill this gap, this invention establishes the most comprehensive benchmark platform currently available, covering 72 specific tasks across 6 different clinical task types, including slice-level classification, survival prediction, region of interest (ROI)-tissue classification, ROI retrieval, visual question answering, and report generation, to evaluate the performance of off-the-shelf foundational models. Research has found that existing foundational models perform well on specific task types but struggle to effectively handle the full range of clinical tasks. To improve the generalization ability of pathology foundational models, this invention proposes a unified knowledge distillation framework, comprising expert knowledge distillation and self-knowledge distillation—the former enabling the model to learn knowledge from multiple expert models, and the latter achieving local-global aligned image representation learning through self-distillation. Based on this framework, this invention constructed a dataset containing 96,000 whole-slice images (WSIs) and developed a Generalized Pathology Foundation Model (GPFM). This state-of-the-art model was trained on a massive dataset of 190 million images, comprising approximately 72,000 publicly available slides and covering 34 major tissue types. In the established benchmark tests, GPFM achieved an excellent average ranking of 1.6, ranking first in 42 tasks; while the second-ranked UNI model had an average ranking of 3.7, ranking first in only 6 tasks. GPFM's superior generalization ability demonstrates its outstanding modeling capabilities across various clinical tasks, making it a new cornerstone for feature representation in computational pathology.

[0112] In some specific implementations, as shown in Figure 3, an intuitive approach to improve the generalization ability and overall performance of the basic pathology model is to utilize knowledge distillation techniques, leveraging the specific advantages of existing models. Accordingly, this invention proposes a novel self-supervised learning framework that combines expert knowledge distillation and self-knowledge distillation to develop a general pathology basic model (GPFM). Based on the aforementioned pre-training method, this invention collected a dataset of 95,572 slides containing 34 major tissue types for GPFM training and evaluation. From this dataset, this invention extracted 190 million image patches from 72,280 slides for pre-training (see Figure 3, a. labeled portion). With the aid of the diverse tissue data collected, and by indirectly using images used for pre-training expert models (such as UNI and CONCH), GPFM demonstrated superior performance in established benchmark tests (see Figure 3, b. and c. labeled portions), achieving an average ranking of 1.6, while the second-ranked UNI model had an average ranking of 3.7 (see Figure 3, c. labeled portion). These results demonstrate the effectiveness of GPFM as a general foundational model in computational pathology, showcasing its potential to significantly advance the field. The stable performance of GPFM across various clinical tasks highlights the benefits of integrating the strengths of specialized expert models using knowledge distillation. This approach contributes to the development of more robust and functional foundational models (FMs), thereby enhancing their practicality in supporting clinical decision-making and improving patient care outcomes.

[0113] Specifically, GPFM is a state-of-the-art pre-trained base model that demonstrates superior performance across 72 diverse tasks. As shown in Figure 3, a. labeled section: The GPFM dataset contains a large-scale dataset of 95,572 slices covering 34 major tissue types, supporting comprehensive model training and evaluation. b. and c. labeled sections: Performance evaluation of the base models (FMs) across various tasks: 52 internal tasks and 20 external tasks. Only the top 4 models are shown here. d. labeled section: Overview of the unified knowledge distillation process in GPFM. Experts used for expert knowledge distillation are selected based on their average performance across 6 different clinical tasks. The pre-training algorithm comprises three key components: 1) Masked Image Modeling (MIM), 2) Self-Distillation, and 3) Expert Knowledge Distillation. GPFM parameters are updated using an exponential moving average (EMA). Once the model is trained, the full-slice image is first split, and then features are extracted from the split patches. Based on these extracted features, various downstream tasks can be performed, such as benign / malignant classification, lesion detection, and report generation, depending on the task requirements.

[0114] In some specific embodiments, the general pathology task processing method of the present invention can be implemented through the following process steps:

[0115] 1. Dataset preparation:

[0116] To improve the performance of the base model, diverse tissue datasets are crucial. This invention collected over 33 datasets. For processing full-slice images, this invention employed the OpenSlide and CLAM toolkits to extract all non-overlapping 512×512 image patches containing tissues at level 0 resolution. Notably, this invention did not scale the image patches to a uniform resolution, but rather preserved the original resolution of each full-slice image. This approach was adopted to improve the robustness of the base model to different resolutions. For datasets containing only regions of interest (ROI) images, this invention also extracted non-overlapping 512×512 image patches. After processing all 33 datasets, this invention obtained a comprehensive dataset. The pre-training data includes 72,280 full-slice images and a total of 190,212,668 image patches.

[0117] 2. Pre-training combining self-knowledge distillation and expert knowledge distillation:

[0118] In the field of computational pathology, existing foundational models typically rely on state-of-the-art self-supervised pre-training (SSL) methods such as DINOv2 and iBOT. These methods are directly applied to private or public datasets. For example, Phikon is built using iBOT based on 6,093 TCGA slices, while UNI is built using DINOv2 based on approximately 100,000 private and public slices. Due to its larger training dataset and the greater power of the SSL method, UNI outperforms Phikon in various tasks. However, due to its limited pre-training strategy and primary site coverage, UNI still lags behind other foundational models in text analysis and survival analysis-related tasks. To address the limitations of existing foundational models and further improve their performance, this invention first proposes a novel pre-training strategy—Unified Knowledge Distillation. The proposed pre-training framework is similar to DINOv2. This invention employs a teacher-student network, combining Masked Image Modeling (MIM) loss and DINO (Self-Distillation) loss to optimize the student network. Specifically, given an input image x, two augmented views u and v are obtained. Then, u and v are randomly masked to obtain masked views. and For MIM objectives, student networks with and As input, the goal is to predict the masked labels. For the DINO objective, this invention first prunes n additional local views wi and uses a student network to extract the encoded class ([CLS]) labels. Next, this invention uses a teacher network to obtain the [CLS] labels for the global views (u and v). Finally, the cross-entropy loss between the local and global view [CLS] labels is calculated. However, this strategy fails to leverage the knowledge of existing visual foundational models (such as UNI) and visual-language foundational models (such as CONCH), limiting its applicability across different tissue types. To facilitate the transfer of knowledge from established pathological foundational models, this invention proposes an expert knowledge distillation module designed to distill knowledge into the student network. To maximize the generalization of the pre-trained model, balancing the performance and diversity of the expert model is crucial. This invention evaluates multiple existing models across six different tasks and selects models that perform well in classification (UNI), report generation (Phikon), and visual question answering (CONCH) as expert models. The [CLS] labels, representing the overall information of an image patch, are used for downstream tasks and are a key component of the method in this invention. If the [CLS] tags of the model of this invention are highly aligned with the [CLS] tags of the expert model, it indicates that the model of this invention can effectively absorb the knowledge of the selected expert. Similarly, the [PATCH] tags also contain rich information. For example, some methods use average pooling to perform downstream tasks. Therefore, aligning the [PATCH] tags can further improve the effect of knowledge transfer. To achieve the above alignment, this invention uses a student network to encode the global view and extract the [CLS] and [PATCH] tags. In addition, this invention uses the expert models employed to obtain their [CLS] and [PATCH] tags respectively. For class tag alignment, this invention uses cosine similarity. For image patch tag alignment, this invention uses both cosine similarity and smooth L1 distance (SmoothL1). The pseudocode for this process is shown in Algorithm 1 in Table 1 below. After the student network is updated, this invention uses exponential moving average (EMA) to update the teacher network (GPFM).

[0119] Table 1

[0120] In specific application scenarios, this invention evaluated various base models across 72 tasks, including 36 whole-slice image classification tasks, 15 survival analysis tasks, 16 image patch-level (ROI) tissue classification tasks, 2 pathology visual question answering tasks, 2 report generation tasks, and 1 pathology image retrieval task (see the marked parts e. to g. in Figure 4). Since these tasks involve different types of evaluation metrics, this invention uses an average ranking method to evaluate the overall performance of the base models and reports the critical difference (CD) plot. The best-performing model is ranked first, and the worst-performing model is ranked ninth. Across all tasks, the GPFM model achieved the highest average ranking score of 1.6 (ranked first in 42 tasks), outperforming the second-ranked UNI model (ranked 3.7, ranked first in 6 tasks). To assess the significance of GPFM's ranking score relative to other base models, this invention performed the Nemenyi statistical test (marked part d. in Figure 4). The results show that GPFM exhibits a statistically significant critical difference compared to the other 8 models. This invention calculated the average evaluation index for all 72 tasks (marked in b. of Figure 4) and found that GPFM achieved the highest average score of 0.833, surpassing the second-ranked UNI model (score 0.818). To assess statistical significance, this invention conducted a Wilcoxon signed-rank two-tailed test on GPFM and the second and third-ranked models. The results show that all p-values ​​are below 0.001, confirming that GPFM consistently and significantly outperforms existing base models. From both ranking and average index perspectives, the results clearly demonstrate that GPFM achieves state-of-the-art performance and generalizes better than other base models.

[0121] Figure 4 shows a comprehensive comparison of various base models across 72 tasks. Figure 4a. Marked section: Task types evaluated by different base models. Figure 4b. Marked section: Average performance of various base models across 72 tasks: AUC (Area Under the Curve) is used for whole-slice image classification and tissue classification tasks; C-index is used for survival analysis tasks; overall accuracy is used for visual question answering tasks; average metrics of BLUE, METEOR, and ROUGE-L are used for report generation tasks; and average accuracy is used for image retrieval tasks. A Wilcoxon signed-rank two-tailed test was used to detect significant differences between existing base models and the proposed GPFM. The error bars in b and c represent 95% confidence intervals, indicating that GPFM achieved the highest average performance. Figure 4c. Marked section: Average ranking of various base models across 72 downstream tasks. Figure 4d. Marked section: Critical difference (CD) plot of average ranking scores using the Nemenyi test. In the CD plot, there are no significant differences between models covered by the black line. Figure 4, marked e. and f., shows the ranking of the various base models across 32 and 20 internal tasks. Figure 4, marked g., shows the ranking of the various base models across 20 external validation datasets. A model is assigned a ranking value of 1 if it performs best. If two models have the same metric value (i.e., tied), an average rank value is assigned to all tied models. For Full Slice Image-Visual Question Answering (WSI-VQA), the ranking is determined by the average language evaluation metric and closed-ended accuracy. The evaluation metrics used for the ranking scores of the remaining tasks are consistent with those marked b. in Figure 4.

[0122] The following examples illustrate the effects of GPFM for different types of tasks:

[0123] (1) Classification of whole-slice images:

[0124] Full-slice image classification is crucial for accurate cancer diagnosis. It helps classify specific subtypes of cancer, and utilizing a base model can significantly improve this process. Therefore, evaluating the representation learning capabilities of different base models is of great significance. As shown in Figure 5, experiments were conducted on a total of 36 tasks, including 20 internal validation datasets and 16 external validation datasets. In the 36 full-slice image classification tasks, based on the AUC (Area Under the Curve) metric, GPFM achieved an excellent average ranking score of 1.22, significantly outperforming the second-ranked UNI model (average ranking score of 3.60) (Figure 5, a. marked portion). This invention uses average metrics (AUC, balanced accuracy, and weighted F1 score) to evaluate overall performance. Specifically, GPFM achieved the highest average AUC of 0.891, a 1.6% improvement over UNI (P < 0.001; Figure 5, d. marked portion). Similarly, GPFM also outperforms UNI in both balanced accuracy (0.752, +3.1%, P<0.001; Figure 5 b. labeled portion) and weighted F1 score (0.736, +3.0%, P<0.001; Figure 5 c. labeled portion). Furthermore, GPFM performs best on both internal and external tasks, with AUCs of 0.938 (1.6% higher than UNI, Figure 5 e. labeled portion) and 0.832 (1.5% higher than UNI, Figure 5 f. labeled portion), respectively. These results across multiple metrics highlight GPFM's strong generalization ability and its potential in whole-slice image classification tasks.

[0125] GPFM improves diagnostic accuracy across multiple cancer types. GPFM demonstrates superior diagnostic accuracy across various cancer types and tasks. In breast cancer, GPFM outperforms other models in all six assessment tasks, including five subtype classification tasks (marked j in Figure 5) and one metastasis detection task. In lung cancer, GPFM excels in three subtype classification tasks, two metastasis detection tasks, and two primary site prediction tasks, with UNI performing only slightly better in external validation for lung cancer metastasis detection. In gastric cancer, GPFM is best-performing in six of the nine tasks, including vascular invasion detection (marked h in Figure 5), perineural invasion detection, and Lauren subtype classification. Furthermore, GPFM consistently demonstrates top performance in tasks involving other organs, such as brain tumor subtype classification, head and neck cancer primary site and T-staging prediction, colon lesion grading, prostate cancer grading assessment, ovarian cancer subtype classification, and renal cell carcinoma classification (marked g in Figure 5). Overall, GPFM establishes its leading model position in cancer diagnosis across various tasks and cancer types.

[0126] GPFM advances gene mutation prediction. Experiments were conducted on lung cancer and brain cancer slides. GPFM achieved state-of-the-art results in both lung cancer TP53 gene mutation prediction (AUC 0.855, 1.3% higher than Phikon) and glioma IDH1 gene mutation prediction (internal AUC 0.986, external AUC 0.943). These results, along with findings in cancer diagnosis, highlight GPFM's superior generalization compared to existing base models. A key factor in its success is GPFM's ability to integrate knowledge from expert models through a unified knowledge distillation mechanism. Unlike previous base models that did not employ knowledge distillation, GPFM leverages this method to learn from a wider range of data and perspectives, significantly improving its performance. This capability underscores GPFM's advanced adaptability and effectiveness across various tasks.

[0127] Figure 5 shows the performance of various base models in whole-slice image classification tasks. Figure 5a. Labeled section: AUC based on 36 whole-slice image classification tasks, and the average ranking of various base models. Figure 5b to d. Labeled sections: Average balanced accuracy (ACC), weighted F1 score (F1), and AUC of various base models in 36 whole-slice image classification tasks. Figure 5e. Labeled section: Average AUC of various base models in 20 internal whole-slice image classification tasks. Figure 5f. Labeled section: Average AUC of various base models in 16 external validation cohorts. Figure 5g to h. Labeled sections: Model performance on specific tasks: renal cell carcinoma (RCC) subtype classification, vascular invasion detection, ovarian cancer subtype classification, and breast cancer subtype classification. * represents external validation cohorts. Error bars represent 95% confidence intervals.

[0128] (2) Survival analysis:

[0129] Figure 6 shows the performance of various base models across 15 survival analysis tasks. Figure 6a. Marked section: Average ranking of various base models across the 15 survival analysis tasks. Figure 6b. Marked section: Average C-index of various base models across the 15 tasks. Figure 6c. Marked section: Results on the TCGA-HNSC dataset and the HANCOCK cohort. Survival prediction models were trained on the TCGA-HNSC cohort and subsequently tested on the HANCOCK cohort. Figure 6d to f. Marked sections: C-index of various base models across 12 survival analysis tasks. In all marked sections, error bars represent 95% confidence intervals. For box plots, the center line represents the mean, and the box plot boundaries represent the standard error.

[0130] Accurate prediction of patient survival risk helps in developing more targeted and effective treatment strategies. Robust base models are crucial for improving the accuracy of survival risk prediction, ultimately improving patient prognosis. To evaluate the performance of various base models in survival analysis, this invention conducted experiments on 15 datasets. Following previously proposed methods, this invention uses the C-index as an evaluation metric to compare the performance of different base models. In the 15 survival analysis tasks, GPFM achieved an excellent average ranking score of 2.1, ensuring best or second-best performance in 13 tasks (marked in Figure 6 a. and d. to f.). In contrast, the second-ranked UNI model had an average ranking score of 4.6, achieving top-two performance only in 4 tasks (marked in Figure 6 a. and d. to f.). Furthermore, GPFM performed best according to the widely accepted C-index metric, with an average C-index of 0.665 (marked in Figure 6 b.). This result is a significant improvement of 3.4% over UNI (P<0.001), further demonstrating GPFM's superior generalization ability in survival analysis tasks. To further validate the generalization ability of the base model, this invention conducted additional validation studies, including one external validation for head and neck cancer (TCGA-HNSC) and one internal validation for lung adenocarcinoma (TCGA-LUAD). In the head and neck cancer survival prediction task, UNI performed best in both the TCGA-HNSC and HANCOCK cohorts, while the method of this invention ranked second (marked in Figure 6, c). However, in the lung adenocarcinoma task, GPFM improved the C-index by 10.6% compared to UNI in the CPTAC-LUAD cohort.

[0131] It is worth noting that survival analysis is inherently more challenging than whole-slice image classification, and no single model can dominate these tasks. Experimental results for both whole-slice image classification and survival analysis highlight the limited generalization ability of existing base models. This limitation can be attributed to the data distribution of their training sets and the pre-training methods employed. Despite their limited generalization ability, existing base models exhibit superior performance in specific types of tasks. By leveraging their respective strengths, it is possible to build more powerful and feature-rich models. This is precisely what this invention achieves: it proposes a unified distillation framework that distills the capabilities of existing models (especially in tasks they excel at) into GPFM, thereby significantly improving their generalization ability.

[0132] (3) Region of Interest (ROI) Classification:

[0133] Figure 7 shows the performance of various base models in tissue classification tasks. Figure 7a. Labeled section: AUC based on 16 tasks, average ranking order of various base models. Figure 7b to d. Labeled sections: Average balanced accuracy (ACC), weighted F1 score (F1), and AUC of various base models in 16 tasks. The center line represents the mean, and the box plot boundaries represent the standard error. Figure 7e to i. Labeled sections: AUC of various base models in 5 tissue classification tasks. Significant differences were detected using the Wilcoxon signed-rank one-sided test. The black center line in the violin plot represents the mean AUC. Figure 7j. Labeled section: Tumor-infiltrating lymphocyte classification based on pan-cancer-TIL (internal) and Center-3-TIL data (external). Figure 7k. Labeled section: Gastric cancer tissue classification based on GasHisDB (internal) and Center-3-GC data (external). In all subplots, error bars represent 95% confidence intervals.

[0134] The performance of full-slice image classification is affected by the feature extractor (i.e., the base model) and the multiple instance learning (MIL) method. Unlike full-slice image classification, region of interest (ROI) classification tasks can directly evaluate the feature representation ability of the base model without being affected by the MIL method. Therefore, this invention employs a linear probing method to evaluate the base model. The evaluation of this invention covers 16 ROI classification tasks, including 13 internal and 3 external validation datasets. GPFM performs best across all 16 ROI classification tasks, achieving a best ranking score of 1.88, significantly outperforming the second-ranked Prov-Gigapath model (score 3.09) (Figure 7a. marked portion). In terms of traditional metrics, GPFM achieved the highest mean AUC of 0.946 (0.2% higher than Prov-Gigapath, P<0.001; d. labeled in Figure 7), the best weighted F1 score of 0.865 (+0.9%, P<0.001; c. labeled in Figure 7), and the highest balanced accuracy of 0.866 (+1%, P<0.001; b. labeled in Figure 7). GPFM performed well on multiple tasks, including breast cancer metastasis detection (g. labeled in Figure 7), lung cancer tissue type classification (h. labeled in Figure 7), tumor-infiltrating lymphocytes (TILs) classification (j. labeled in Figure 7), and gastric cancer tissue classification (k. labeled in Figure 7). In the relatively simple region of interest classification task, GPFM shared top rankings with other base models. For example, in pan-cancer tissue classification, breast tumor classification, colorectal cancer tissue classification (Figure 7 f. marked portion), and kidney tissue classification (Figure 7 e. marked portion), GPFM achieved performance comparable to other leading base models. In tasks where GPFM did not achieve the best performance, it consistently ranked second (Figure 7 i. marked portion) or third. This consistently high ranking across various tasks contributes to GPFM's overall superior performance. Furthermore, the average ranking scores for UNI and Prov-Gigapath (Figure 7 a. marked portion) are very close, at 3.2 and 3.1 respectively. This indicates that no single existing model dominates in region of interest classification tasks. In contrast, by integrating the knowledge of all employed expert models, unified knowledge distillation enabled GPFM to surpass the performance of individual models, achieving a significantly lower average ranking score of 1.88, more than one point higher than the second-ranked model. This highlights GPFM's advantage as a highly general base model. Furthermore, to evaluate GPFM's robustness in processing images at different resolutions, this invention visualizes a heatmap of attention scores between the [PATCH] and [CLS] markers of the ViT converter.The invention observed a consistent attention pattern across four resolutions (224×224, 448×448, 896×896, and 1344×1344), highlighting the robustness of GPFM in adapting to different image resolutions.

[0135] (4) Pathological image retrieval:

[0136] Figure 8 provides an overview of pathological tissue retrieval, visual question answering, and report generation. It should be noted that in practical applications, the language of the generated report can be adjusted according to actual needs. The example in Figure 8 uses an English report and should not be considered a limitation of the embodiments of this invention. Furthermore, the report content in Figure 8 is merely illustrative and does not limit the principles or logic of the technical solutions of the embodiments of this invention. Figure 8 shows: a. Marked section: Top-1, Top-3, Top-5, and average accuracy of different base models in the pathological tissue retrieval task. Figure 8 shows: b. Marked section: Feature distribution extracted by GPFM. For each category, 100 samples from the test set, totaling 900 samples, were used for t-SNE dimensionality reduction to two dimensions. Figure 8 shows: c. Marked section: Visual question answering performance of different base models on the PathVQA dataset, measured using open-ended accuracy, closed-ended accuracy, and overall accuracy. Figure 8 shows: d. Marked section: An open-ended question and the answers generated by various base models. Figure 8 shows: e. Marked section: Three questions related to the query image and the answers generated by the base models. Figure 8, section f.: Report generation performance of various base models on TCGA and PATCHGastricAD22 data. Six different metrics are used to measure model performance. In all subplots, error bars represent standard deviation.

[0137] Image retrieval technology can match new patient pathology images with a carefully curated database of previously confirmed cases, providing pathologists with a novel tool to improve diagnostic accuracy. By visually examining and comparing similar historical cases, pathologists can leverage image search capabilities to enhance their diagnostic decisions. In this study, the CRC-100K dataset was used for the pathology image retrieval task. Experimental results (Figure 8a. marked portion) show that the GPFM model achieved the second-best Top-1 accuracy of 0.906 (1.9% lower than Prov-Gigapath). However, GPFM outperformed other models in Top-3 and Top-5 accuracy, reaching 0.993 (0.5% higher than Prov-Gigapath) and 0.995 (0.2% higher than Prov-Gigapath), respectively. To further explore clustering effects and feature representation capabilities, this invention utilizes t-distributed random neighborhood embedding (t-SNE) to project the features extracted by GPFM into a two-dimensional embedding space. The categorical clustering effect was good, further demonstrating the high discriminative power of these features (Figure 8b. marked portion). This invention also visualizes the feature distributions of other base models. GPFM extracts more tightly clustered features, and the query image also lies within the candidate clusters, indicating better clustering performance. This observation demonstrates GPFM's superior feature representation capabilities in capturing the inherent patterns and structures in data.

[0138] (5) Visual Question Answering (VQA) of Pathological Images:

[0139] Visual question answering (VQA) is an exciting direction in artificial intelligence, aiming to enable machines to answer questions about visual content. In pathology, VQA systems can be particularly powerful, allowing clinicians and researchers to quickly and accurately extract relevant information from medical images. For image patch-level VQA tasks, the model of this invention achieved second-best performance, only slightly below CONCH (marked portion c in Figure 8). Notably, CONCH is a visual-language foundation model trained on millions of image-text pairs, giving it an inherent advantage in VQA tasks. Nevertheless, the results of this invention highlight the significant potential of the method compared to other purely visual foundation models. To further illustrate the capabilities of the model of this invention, the query images, questions, and answers generated by different foundation models are visualized (marked portions d and e in Figure 8). Compared to other models, GPFM and CONCH consistently generate more reliable and accurate answers. Furthermore, in the full-slice image-level VQA task, the model of this invention achieved best or second-best performance in 6 out of 7 metrics, performing comparably to the slice-level foundation model CHIEF. These results, along with the findings at the image patch level, highlight the effectiveness of unified knowledge distillation. Specifically, the knowledge gleaned from millions of image-text pairs by CONCH can be successfully distilled into GPFM without accessing the original image-text pair data. The robust performance of GPFM highlights the potential for indirectly utilizing textual knowledge without directly using textual data, thus providing a promising direction for future research in visual question answering tasks.

[0140] (6) Pathology report generation:

[0141] Pathology reports are a vital component of the healthcare system, providing clinicians and patients with crucial information on the diagnosis, prognosis, and treatment of various medical conditions. These reports summarize the results of pathological examinations, such as biopsies, cytological samples, and surgical specimens, and play a critical role in guiding clinical decisions. Traditionally, pathology reports are manually prepared by pathologists and their teams, a time-consuming and laborious process. Recent advances in Natural Language Processing (NLP) and machine learning have enabled the development of automated pathology report generation systems, which can significantly improve the efficiency and consistency of this critical task. To evaluate the effectiveness of the underlying models in this field, this invention assesses their performance on the TCGA whole-slice image-report dataset compiled by Guo et al. and the PATCHGastricADC22 dataset.

[0142] Experimental results show that Phikon performs best across all six metrics, while GPFM achieves comparable performance, ranking second in both tasks (f. marked in Figure 8). Surprisingly, visual foundational models (such as Phikon and GPFM) outperform visual-language foundational models (such as CONCH and PLIP) significantly in this task. This performance gap may be attributed to the training paradigm of PLIP and CONCH—they rely solely on brief descriptions or titles of pathological images, failing to capture global contextual information. Therefore, these text-image pairs perform poorly in full-report generation compared to their application in the original visual question-answering task. To further validate these findings, this invention performs a hierarchical report generation analysis using the TCGA full-slice image-report dataset, stratified by major cancer types (i.e., breast cancer, lung cancer, and kidney cancer), for independent evaluation. The results show that Phikon maintains its advantage in breast and lung cancer report generation but slightly lags behind the GPFM of this invention in kidney cancer report generation. To leverage the complementary strengths of existing foundational models, the proposed unified knowledge distillation method can distill Phikon's capabilities in report generation into GPFM. This synergistic integration allows the present invention to combine the strengths of these foundational models to develop a more generalizable model. To further assess clinical relevance, an experienced pathologist evaluated the diagnostic reports using a four-level scoring system. Blinded human evaluation results showed that GPFM performed exceptionally well, achieving the highest average scores in breast, lung, and kidney cancer reports. These expert-validated results highlight the potential of the present invention's unified knowledge distillation method to generate clinically meaningful reports that meet pathologist diagnostic criteria, marking a significant step forward in the practical application of artificial intelligence in automating pathology workflows.

[0143] The effectiveness analysis of expert knowledge distillation in some specific application scenarios is as follows:

[0144] Figure 9 illustrates an example of evaluating the effectiveness of expert knowledge distillation, showing the performance difference between GPFM (with expert knowledge distillation) and DINOv2 (without expert knowledge distillation). The black horizontal line represents the mean AUC. If GPFM outperforms DINOv2, the p-value is also reported. Figure 9 shows: a. Marked section: Balanced accuracy of the models with and without expert knowledge distillation. Figure 9 also shows: b. Marked section: Weighted F1 score of the models with and without expert knowledge distillation. Figure 9 further shows: c. Marked section: AUC of the models with and without expert knowledge distillation. The center line represents the mean, and the dashed lines represent the 2.5 percentile and 97.5 percentile. Significance was tested using the Wilcoxon signed-rank one-sided test, and the results show that expert knowledge distillation consistently improves performance in most tasks, highlighting the effectiveness of this technique in enhancing GPFM.

[0145] In the self-supervised learning framework proposed in this study, a unified knowledge distillation model is introduced to facilitate knowledge transfer from off-the-shelf base models to GPFM during the pre-training phase. To evaluate the effectiveness of this module, an experiment was conducted—removing the expert knowledge distillation module to obtain a modified self-supervised learning framework, termed DINOv2. DINOv2 and GPFM were trained on the same dataset, and their performance was evaluated on tissue classification tasks. The experimental results clearly demonstrate that expert knowledge distillation has a positive impact on model performance across 12 tasks. The results show significant improvements not only in individual task performance but also in overall average performance—all three evaluation metrics show substantial improvements: AUC increased by 0.6%, weighted F1 score increased by 1.8%, and balanced accuracy increased by 1.8%. These findings provide strong evidence for the effectiveness of transferring knowledge from off-the-shelf pathology base models using the proposed knowledge distillation learning framework. However, even after distillation, GPFM still fails to outperform the original DINOv2 on all tasks (such as Chaoyang and BreakHis), indicating that there is still room for improvement in the distillation strategy.

[0146] In summary, this invention constructs the most comprehensive benchmark platform for computational pathology tasks to date. Furthermore, this invention introduces GPFM—a general-purpose foundational model designed for various computational pathology tasks. To enhance the model's versatility, this invention proposes a unified knowledge distillation pre-training framework, which effectively integrates expertise from various existing models. This innovative approach ensures that GPFM can adapt and excel in diverse computational pathology tasks. To further maximize the diversity of pre-training data, this invention collects 190 million images from 56 sources, covering 34 major tissue types. This rich dataset, combined with the invention's advanced pre-training methods, enables GPFM to outperform existing foundational models in 72 computational pathology tasks. Unlike other models that excel only in narrow domains—such as UNI, which excels in whole-slice image classification, and Phikon, which excels in report generation—GPFM demonstrates superior generalization ability by combining the advantages of expert models, outperforming its counterparts in various computational pathology challenges.

[0147] This invention also provides a universal pathology task processing device that can implement the above-described method. This device may include:

[0148] The first module is used to obtain the tissue dataset; the data samples of the tissue dataset include image patches corresponding to full-slice images;

[0149] The second module is used to process data samples using a pre-defined student model to obtain global and local feature labels for students.

[0150] The third module is used to obtain expert global feature labels and expert local feature labels based on data samples and using the expert model corresponding to the preset pathology task.

[0151] The fourth module is used to construct class label alignment loss based on student global feature labels and expert global feature labels, and to construct image patch label alignment loss based on student local feature labels and expert local feature labels;

[0152] The fifth module is used to sum up the class label alignment loss and the image patch label alignment loss to obtain the total loss;

[0153] The sixth module is used to configure the teacher network based on the total loss and the teacher-student network strategy.

[0154] Module 7 is used to process pathology tasks using a teacher network.

[0155] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0156] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0157] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0158] As shown in Figure 10, Figure 10 illustrates the hardware structure of an electronic device 1000 according to another embodiment. The electronic device 1000 includes:

[0159] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0160] The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001.

[0161] Input / output interface 1003 is used to implement information input and output;

[0162] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0163] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0164] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0165] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0167] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0168] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0169] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0170] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0171] The general pathology task processing method, apparatus, electronic device, storage medium, and program product provided in this invention involve acquiring a tissue dataset; wherein the data samples of the tissue dataset include image patches corresponding to whole-slice images; based on the data samples, processing is performed using a preset student model to obtain student global feature labels and student local feature labels; based on the data samples, processing is performed using a preset expert model corresponding to the pathology task to obtain expert global feature labels and expert local feature labels; a class label alignment loss is constructed based on the student global feature labels and expert global feature labels, and an image patch label alignment loss is constructed based on the student local feature labels and expert local feature labels; the class label alignment loss and image patch label alignment loss are summed to obtain a total loss; based on the total loss, a teacher network is configured using a teacher-student network strategy; and the pathology task is processed using the teacher network. This invention integrates expert model knowledge by pre-setting expert models corresponding to pathology tasks, enabling the construction of a general pathology foundation model and avoiding repeated training for each task. Furthermore, this invention utilizes a dual alignment loss of class labels and image patch labels, allowing student models to directly learn expert features, effectively improving training efficiency and performance. Additionally, this invention employs a teacher-student network and an exponential moving average strategy to enhance model convergence stability and generalization ability, thereby supporting the subsequent expansion of new expert models. This invention provides a complete process from data preparation to task processing, reducing system integration and deployment complexity.

[0172] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0173] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0176] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A general method for processing pathological tasks, characterized in that, The method includes the following steps: Obtain a tissue dataset; wherein, the data samples of the tissue dataset include image patches corresponding to full-slice images; Based on the data sample, the global feature labels and local feature labels of students are obtained by using a preset student model. Based on the data sample, expert global feature labels and expert local feature labels are obtained by using the expert model corresponding to the preset pathology task. A class label alignment loss is constructed based on the student global feature labels and the expert global feature labels, and an image patch label alignment loss is constructed based on the student local feature labels and the expert local feature labels; The total loss is obtained by summing the class label alignment loss and the image patch label alignment loss. Based on the total loss, a teacher network is configured using a teacher-student network strategy. The aforementioned teacher network was used to process pathology tasks.

2. The method according to claim 1, characterized in that, The process of obtaining the organizational dataset includes the following steps: Collect the raw dataset; wherein the raw dataset includes multiple full-slice images; Based on the original dataset, image patches containing tissue of a preset size are extracted from each full-slice image using a preset image tool, and then the tissue dataset is obtained.

3. The method according to claim 1, characterized in that, The method of constructing class label alignment loss based on the student global feature labels and the expert global feature labels includes the following steps: A first similarity evaluation is performed on the student global feature label and the expert global feature label to obtain the first cosine similarity between the student global feature label and the expert global feature label; The class label alignment loss is constructed based on the complementary value of the first cosine similarity compared to 1.

4. The method according to claim 1, characterized in that, When the preset pathology task includes multiple sub-tasks, the expert global feature label includes the sub-expert global feature label obtained by the expert model processing for each sub-task. The step of constructing a class label alignment loss based on the student global feature label and the expert global feature label includes the following steps: A second similarity evaluation is performed on the student global feature label and each of the sub-expert global feature labels to obtain the second cosine similarity between the student global feature label and each of the sub-expert global feature labels; The subclass label alignment loss for each subtask is constructed based on the complementary value of the second cosine similarity compared to 1. The class label alignment loss is obtained by performing a first weighted summation on the subclass label alignment loss corresponding to all the subtasks.

5. The method according to claim 1, characterized in that, The method of constructing image patch label alignment loss based on the student local feature labels and the expert local feature labels includes the following steps: A third similarity evaluation is performed on the student local feature labels and the expert local feature labels to obtain the third cosine similarity between the student local feature labels and the expert local feature labels; A first distance evaluation is performed on the student local feature label and the expert local feature label to obtain a first smoothed L1 distance between the student local feature label and the expert local feature label; The first image patch loss is constructed based on the complementary value of the third cosine similarity compared to 1, and the second image patch loss is constructed based on the complementary value of the first smooth L1 distance compared to 1. The image block label alignment loss is obtained by performing a second weighted summation on the first image block loss and the second image block loss.

6. The method according to claim 1, characterized in that, When the preset pathology task includes multiple sub-tasks, the expert local feature labels include sub-expert local feature labels obtained by the expert model processing for each sub-task. The step of constructing image patch label alignment loss based on the student local feature labels and the expert local feature labels includes the following steps: A fourth similarity evaluation is performed on the student local feature label and each of the sub-expert local feature labels to obtain the fourth cosine similarity between the student local feature label and each of the sub-expert local feature labels; A second distance evaluation is performed on the student local feature label and each of the sub-expert local feature labels to obtain the second smoothed L1 distance between the student local feature label and each of the sub-expert local feature labels; The third image block loss corresponding to each subtask is constructed based on the complementary value of the fourth cosine similarity compared to 1, and the fourth image block loss corresponding to each subtask is constructed based on the complementary value of the second smooth L1 distance compared to 1. The third image block loss and the fourth image block loss of each subtask are summed by a third weighted sum to obtain the sub-image block label alignment loss of that subtask. The image patch label alignment loss is obtained by performing a fourth weighted summation on the sub-image patch label alignment loss corresponding to all the sub-tasks.

7. The method according to claim 1, characterized in that, The process of configuring the teacher network based on the total loss using a teacher-student network strategy includes the following steps: Based on the total loss, the weight parameters of the student model are updated through backpropagation; Based on the updated weight parameters of the student model, the weights of the preset teacher network are updated using an exponential moving average to obtain the target teacher network.

8. A universal pathology task processing device, characterized in that, The device includes: The first module is used to acquire a tissue dataset; wherein, the data samples of the tissue dataset include image patches corresponding to full-slice images; The second module is used to process the data sample using a preset student model to obtain global and local feature labels for students. The third module is used to obtain expert global feature labels and expert local feature labels based on the data samples and using the expert model corresponding to the preset pathology task. The fourth module is used to construct a class label alignment loss based on the student global feature labels and the expert global feature labels, and to construct an image patch label alignment loss based on the student local feature labels and the expert local feature labels; The fifth module is used to sum up the class label alignment loss and the image patch label alignment loss to obtain the total loss; The sixth module is used to configure the teacher network based on the total loss using a teacher-student network strategy. The seventh module is used to process pathology tasks using the aforementioned teacher network.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images