Tissue morphology expression device, tissue morphology expression method, and recording medium
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- NEC CORP
- Filing Date
- 2025-01-27
- Publication Date
- 2026-07-30
Smart Images
Figure JP2025002442_30072026_PF_FP_ABST
Abstract
Description
Tissue Morphology Expression Device, Tissue Morphology Expression Method, and Recording Medium
[0001] The present disclosure relates to a technology for verbalizing the morphological image of a pathological tissue.
[0002] Techniques for assisting pathological tissue examinations by analyzing pathological images are known. For example, Patent Document 1 discloses an image diagnosis system that uses an artificial intelligence function to assist in creating a diagnosis report for a pathological image.
[0003] Japanese Patent Application Laid-Open No. 2022-146822
[0004] However, even with the method of Patent Document 1, a diagnosis report may not always be created with appropriate language expression.
[0005] One object of the present disclosure is to provide a tissue morphology expression device that performs a diagnosis of a pathological tissue image with appropriate language expression.
[0006] In one aspect of the present disclosure, the tissue morphology expression device includes: first partial image generation means for generating a partial image smaller than the input image from an input image including the tissue morphology of a target patient; prediction means for inputting the partial image into a learned GNN and predicting the relationship between the tissue morphology included in the partial image and the language expression of the tissue morphology, the relationship between the language expression of the tissue morphology and the case class, and the relationship between the case class and the treatment method; graph generation means for generating a prediction graph based on the prediction result; and language output means for inputting the prediction graph into a large language model and outputting an answer from the large language model.
[0007] In another aspect of the present disclosure, the tissue morphology expression method includes: generating a partial image smaller than the input image from an input image including the tissue morphology of a target patient; inputting the partial image into a learned GNN and predicting the relationship between the tissue morphology included in the partial image and the language expression of the tissue morphology, the relationship between the language expression of the tissue morphology and the case class, and the relationship between the case class and the treatment method; generating a prediction graph based on the prediction result; inputting the prediction graph into a large language model; and outputting an answer from the large language model.
[0008] In yet another aspect of this disclosure, the recording medium records a program that causes a computer to perform the following processes: generate a smaller subimage from an input image including the tissue morphology of a target patient; input the subimage to a trained GNN; predict the relationship between the tissue morphology contained in the subimage and the linguistic representation of the tissue morphology, the relationship between the linguistic representation of the tissue morphology and the case class, and the relationship between the case class and the treatment method; generate a prediction graph based on the prediction results; input the prediction graph to a large-scale language model; and output a response from the large-scale language model.
[0009] According to this disclosure, it will be possible to diagnose pathological tissue images using appropriate linguistic expressions.
[0010] This is a diagram conceptually showing the organizational morphology representation device related to this disclosure. This is a block diagram showing the hardware configuration of the organizational morphology representation device related to this disclosure. This is a block diagram showing the functional configuration of the organizational morphology representation device related to this disclosure. This is an explanatory diagram of the processing of the image splitting unit, image embedding unit, and organizational classification unit. This is an example of the creation of training data. This is an example of the prediction result by the organizational language conversion unit. This is an example of a prediction graph written in DOT language. This is a flowchart of the training process. This is a flowchart of the prediction process. This is a block diagram showing the functional configuration of another organizational morphology representation device related to this disclosure. This is a flowchart of processing by another organizational morphology representation device related to this disclosure.
[0011] Preferred embodiments of this disclosure will be described below with reference to the drawings. <First Embodiment> [Overall Configuration] The tissue morphology representation device of this embodiment predicts the case and optimal treatment from the pathological tissue image of the patient, and outputs the prediction result along with the reason for selecting the optimal treatment (hereinafter also referred to as "reason for selecting the optimal treatment") in appropriate language. The reason for selecting the optimal treatment includes explanations based on tissue morphology, such as "significant variation in the size of cell nuclei."
[0012] When observing pathological tissue images, observers may be unable to recall appropriate linguistic expressions when describing the morphological appearance of the tissue, or they may use linguistic expressions that differ from those of other observers. Such inconsistencies in linguistic expression for similar morphological images make linguistic searching and database creation difficult. In contrast, the tissue morphology representation device of this embodiment can represent tissue morphology using a unified language, which can be used to explain the reasons for selecting the optimal treatment. As a result, the tissue morphology representation device of this embodiment enables the diagnosis of pathological tissue images using appropriate linguistic expressions.
[0013] Figure 1 is a conceptual diagram of an organizational morphology representation device. The organizational morphology representation device 10 includes a learning unit and a prediction unit.
[0014] The learning unit divides pathological tissue images into multiple partial images and trains a predictive model that predicts the relationship between the partial images and the linguistic descriptions of tissue morphology. Here, the learning unit maps the multiple partial images onto a feature space and divides the feature space into multiple regions where the tissue morphology is similar to each other, based on the density of each mapped point. The learning unit then generates training data based on the linguistic descriptions of tissue morphology associated with each region and trains the predictive model. The task of associating the linguistic descriptions of tissue morphology with each region (annotation) is performed by a user, such as a physician. The user selects an appropriate linguistic description from a pre-prepared set of linguistic descriptions of tissue morphology and associates it with each region. In this way, since annotation is performed using pre-prepared linguistic descriptions of tissue morphology, fluctuations in the linguistic descriptions of tissue morphology can be prevented. In addition, since the user does not need to annotate each individual partial image, training data can be easily generated.
[0015] Furthermore, the predictive model of this embodiment learns not only the relationship between partial images and the linguistic description of tissue morphology, but also the relationship between the linguistic description of tissue morphology and case class, and the relationship between case class and optimal treatment. As a result, the predictive model can predict the relationship between the linguistic description of tissue morphology, case class, and optimal treatment from partial images.
[0016] The prediction unit divides the pathological tissue images of the target patient into multiple partial images and performs predictions for each partial image using a prediction model. Based on the prediction results, the prediction unit creates a prediction graph. The prediction graph represents the relationship between the partial images, the linguistic description of the tissue morphology, the case class, and the optimal treatment. The prediction unit inputs the prediction graph into a general-purpose LLM (Large Language Model) and obtains the linguistic description of the optimal treatment and the reason for its selection as a response from the general-purpose LLM. The prediction unit presents the response from the general-purpose LLM to the user. In addition, the prediction unit can also present the user with the prediction graph in addition to the response from the general-purpose LLM.
[0017] Users such as doctors can determine treatment plans by referring to the responses and predictive graphs from the general-purpose LLM (Low-Level Medical Model).
[0018] [Hardware Configuration] Figure 2 is a block diagram showing the hardware configuration of the organizational structure representation device 10. As shown in the figure, the organizational structure representation device 10 includes an interface (I / F) 11, a processor 12, a memory 13, a recording medium 14, a database (DB) 15, and a display unit 16.
[0019] I / F11 performs data input and output with external devices. Specifically, pathological tissue images are input to the tissue morphology display device 10 via I / F11.
[0020] The processor 12 is a computer such as a CPU (Central Processing Unit) and controls the entire organizational morphology representation device 10 by executing a pre-prepared program. The processor 12 may be a GPU (Graphics Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof. The processor 12 performs the learning process and prediction process described later.
[0021] Memory 13 is composed of ROM (Read Only Memory), RAM (Random Access Memory), and the like. Memory 13 is also used as working memory while the processor 12 is executing various processes.
[0022] The recording medium 14 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the tissue morphology representation device 10. The recording medium 14 stores various programs that the processor 12 executes. When the tissue morphology representation device 10 performs various processes, the programs recorded on the recording medium 14 are loaded into the memory 13 and executed by the processor 12. The DB 15 stores image data input through the I / F 11.
[0023] The display unit 16 is, for example, a liquid crystal display and displays the processing results from the organizational structure representation device 10. In addition to the above, the organizational structure representation device 10 may also be equipped with input devices such as a keyboard and mouse for the user to give instructions or input.
[0024] [Functional Configuration] Figure 3 is a block diagram showing the functional configuration of the organizational morphology representation device 10 of the first embodiment. Functionally, the organizational morphology representation device 10 comprises a learning unit 100 and a prediction unit 200.
[0025] (Learning Unit) First, let me explain the learning unit 100. The learning unit 100 includes an image division unit 101, an image embedding unit 102, an tissue classification unit 103, and a GNN learning unit 104.
[0026] During training, a training dataset is provided containing pathological tissue images from multiple patients. Each pathological tissue image is associated with a case class and treatment method. Case classes represent cases such as "invasive cancer," "carcinoma in situ," "lung cancer," and "adenocarcinoma." Treatment methods include the treatments performed on the patient and whether or not they were effective, and treatments include "immune checkpoint inhibitors," "chemotherapy," and "molecular targeted drugs."
[0027] The image splitting unit 101 divides the pathological tissue images included in the training dataset into smaller partial images. The entire pathological tissue image is also referred to as the "whole image" below. Figure 4 is an explanatory diagram of the processing of the image splitting unit 101, the image embedding unit 102, and the tissue classification unit 103. As shown in Figure 4, the image splitting unit 101 divides the whole image WI into multiple partial images PI and outputs them to the image embedding unit 102. In the example in Figure 4, the whole image WI is an image of the patient's pathological tissue, but the image splitting unit 101 only needs to generate partial images for the pathological tissue region, and does not need to generate partial images for the background region. Also, in the example in Figure 4, the image splitting unit 101 divides the whole image WI into multiple partial images PI by grid division, but instead, it may be divided into multiple partial images PI centered on characteristic points in the whole image WI.
[0028] The image embedding unit 102 converts each of the multiple input sub-images PI into a feature vector to perform dimensionality reduction and maps each sub-image PI onto a low-dimensional feature space. Figure 4 shows an example of mapping each sub-image PI to a two-dimensional feature space. In Figure 4, one sub-image PI is mapped as a single white dot in the two-dimensional feature space. In the feature space of Figure 4, individual white dots are too small to be recognized due to scale, but the area that looks like a white cloud is a collection of white dots. The image embedding unit 102 can, for example, input the sub-images PI into a pre-trained model such as ResNet (Residual Neural Networks) or VGG (Visual Geometry Group), and use the output of its intermediate layer as a feature vector. Furthermore, dimensionality reduction algorithms such as UMAP (Uniform Manifold Application and Projection) and t-SNE (t-distributed Stochastic Neighbor Embedding) can be used.
[0029] The tissue classification unit 103 divides the feature space into multiple regions. For example, the tissue classification unit 103 can divide the feature space into multiple regions by Voronoi partitioning. Specifically, the tissue classification unit 103 divides the feature space based on which other points in the feature space are closest to each representative point (generator) in the feature space. The tissue classification unit 103 can select local peaks as representative points where the density of each point mapped to the feature space is locally maximum. In the feature space of Figure 4, areas with higher density of points are represented in white, and there are nine such peaks. The tissue classification unit 103 selects these nine peaks as representative points. Then, as shown in Figure 4, the tissue classification unit 103 divides the feature space into nine regions, A1 to A9. The tissue classification unit 103 may also divide the feature space into multiple regions using the K-Nearest Neighbors method or the Nearest Neighbor method.
[0030] Points mapped to close positions in the feature space indicate that the image features of the corresponding sub-image PI are similar. Therefore, multiple sub-image PIs corresponding to points within the same region represent images of morphologically similar tissues.
[0031] Users, such as physicians, associate the appropriate linguistic expression from a pre-prepared set of linguistic expressions for tissue morphology with each divided region. Examples of these pre-prepared linguistic expressions for tissue morphology include "large nucleus," "glandular tubule," "lymphocyte," and "infiltration." For example, in Figure 4, if many of the partial image PIs belonging to region A9 are images of large nuclei, the user associates region A9 with "large nucleus." Similarly, if many of the partial image PIs belonging to region A9 are images of lymphocytes, the user associates region A9 with "lymphocyte." Users may associate multiple linguistic expressions for tissue morphology with a single region.
[0032] The organizational classification unit 103 outputs the association between each domain and the linguistic representation of the organizational form to the GNN learning unit 104.
[0033] The GNN learning unit 104 generates training data based on the input from the tissue classification unit 103 and trains a machine learning model. A GNN (Graph Neural Network) is used as the machine learning model. Figure 5 shows an example of training data creation. Note that the images and language shown in Figure 5 are actually represented as vectors.
[0034] As shown in Figure 5(A), first, the GNN learning unit 104 associates the linguistic expression of the tissue morphology assigned to each region with the partial image PI belonging to that region. Then, as shown in Figure 5(B), the GNN learning unit 104 associates the partial image PI with the linguistic expression of the tissue morphology, the linguistic expression of the tissue morphology with the case class, and the case class with the treatment method, based on the case class and treatment method of the pathological tissue image that is the source of the partial image PI.
[0035] The GNN learning unit 104 performs the above-described associations on partial image PIs included in pathological tissue images of multiple patients and generates a graph composed of nodes and links. Figure 5(C) is an example of such a graph. Partial image PIs, linguistic descriptions of tissue morphology, case classes, and treatment methods are represented by nodes, and the relationships between nodes are represented by links. The GNN learning unit 104 uses the generated graph as training data.
[0036] The GNN learning unit 104 learns the relationships between nodes in the graph using a GNN. This trained GNN can take an image as input and predict the relationships between tissues, linguistic descriptions of tissue morphology, case classes, and treatment methods contained in the image. The GNN learning unit 104 outputs the trained GNN to the tissue language processing unit 203. The trained GNN will also be referred to as the "predictive model" below.
[0037] In the above configuration, the image splitting unit 101 is an example of a second partial image generation means, the image embedding unit 102 is an example of a second image embedding means, the tissue classification unit 103 is an example of a tissue classification means, and the GNN learning unit 104 is an example of a learning data generation means and a learning means.
[0038] (Prediction Unit) Next, the prediction unit 200 will be described. The prediction unit 200 includes an image segmentation unit 201, an image embedding unit 202, an organization language processing unit 203, and an LLM unit 204. The image segmentation unit 201 and the image embedding unit 202 of the prediction unit 200 perform the same processing as the image segmentation unit 101 and the image embedding unit 102 of the learning unit 100.
[0039] The image splitting unit 201 splits the pathological tissue image of the target patient into smaller partial images and outputs them to the image embedding unit 202. The pathological tissue image of the target patient is an example of an input image that includes the tissue morphology of the target patient.
[0040] The image embedding unit 202 converts each of the multiple input subimages into a feature vector to perform dimensionality reduction. The image embedding unit 202 outputs the feature vectors of the subimages to the organization language unit 203.
[0041] The tissue language processing unit 203 receives a prediction model from the GNN learning unit 104. The tissue language processing unit 203 inputs feature vectors from partial images into the prediction model and predicts the relationship between partial images, linguistic representations of tissue morphology, case class, and optimal treatment. Figure 6 shows an example of the prediction results by the tissue language processing unit 203. As shown in Figure 6, the tissue language processing unit 203 extracts the portion of the entire graph that is related to the pathological tissue image of the target patient. The extracted partial graph will also be referred to as the "prediction graph" below.
[0042] The organizational language conversion unit 203 converts the prediction graph into text and outputs it to the LLM unit 204. The organizational language conversion unit 203 can represent the prediction graph in text by, for example, using the data description language "DOT language" for describing the graph structure. Figure 7 is an example of a prediction graph described in DOT language. In addition to DOT language, the organizational language conversion unit 203 may also represent the prediction graph in text using other representation methods, such as adjacency matrices.
[0043] The LLM unit 204 inputs the prediction graph as a prompt into the general-purpose LLM. Then, the LLM unit 204 obtains the answer from the general-purpose LLM. The answer from the general-purpose LLM includes a proposal for the optimal treatment and the reason for selecting the optimal treatment. Also, the reason for selecting the optimal treatment includes an explanation based on the tissue morphology. For example, as shown in FIG. 7, the LLM unit 204 inputs a prediction graph into the general-purpose LLM and can obtain, as an answer from the general-purpose LLM, "The size variation of the cell nuclei is remarkable, and infiltration of lymphocytes into the tumor region is observed. Monotherapy with an immune checkpoint inhibitor or combined use of chemotherapy is effective." The LLM unit 204 outputs the answer from the general-purpose LLM to the display unit 16. Also, the LLM unit 204 may output the prediction graph to the display unit 16 together with the answer from the general-purpose LLM.
[0044] Note that, for the general-purpose LLM, GPT (Generative Pre-trained Transformer) of OpenAI etc. can be used.
[0045] In the above configuration, the image segmentation unit 201 is an example of the first partial image generation means, the image embedding unit 202 is an example of the first image embedding means, the tissue languageization unit 203 is an example of the prediction means and the graph generation means, and the LLM unit 204 is an example of the language output means.
[0046] [Learning Process] Next, the learning process will be described. FIG. 8 is a flowchart of the learning process by the tissue morphology expression device 10. This process is realized by the processor 12 shown in FIG. 2 executing a program prepared in advance and operating as each element shown in FIG. 3.
[0047] First, the image segmentation unit 101 divides the pathological tissue image included in the learning dataset into partial images smaller than the pathological tissue image (step S101). Next, the image embedding unit 102 converts each of the partial images into a feature vector and maps it onto the feature space (step S102). Next, the tissue classification unit 103 divides the feature space into a plurality of regions (step S103). The tissue classification unit 103 divides the feature space based on which mother point in the feature space the other points in the feature space are closest to for each representative point (mother point) in the feature space.
[0048] Next, the GNN learning unit 104 obtains the association between each divided region and the linguistic expression of the tissue form, and generates learning data (step S104). Next, the GNN learning unit 104 performs GNN learning using the learning data, and outputs a learned GNN (prediction model) (step S105). Then, the learning process ends.
[0049] [Prediction Process] Next, the prediction process will be described. FIG. 9 is a flowchart of the prediction process by the tissue form expression device 10. This process is realized by the processor 12 shown in FIG. 2 executing a program prepared in advance and operating as each element shown in FIG. 3.
[0050] First, the image segmentation unit 201 divides the pathological tissue image of the target patient into partial images smaller than the pathological tissue image (step S201). Next, the image embedding unit 202 converts each of the input partial images into a feature vector to perform dimensionality reduction (step S202).
[0051] Next, the tissue language conversion unit 203 inputs the feature vector of the partial image into the prediction model, predicts the relationship between the partial image, the linguistic expression of the tissue form, the case class, and the optimal treatment, and generates a prediction graph (step S203). The tissue language conversion unit 203 converts the prediction graph into text and outputs it to the LLM unit 204.
[0052] Next, the LLM unit 204 inputs the prediction graph as a prompt into the general-purpose LLM, and obtains the answer of the general-purpose LLM (step S204). The LLM unit 204 outputs the answer of the general-purpose LLM and the prediction graph (step S205). Then, the prediction process ends.
[0053] [Modification Example] Next, a modification example of the first embodiment will be described.
[0054] The GNN learning unit 104 generates training data by associating partial images with linguistic descriptions of tissue morphology, linguistic descriptions of tissue morphology with case classes, and case classes with treatment methods. In addition, the GNN learning unit 104 may also include the association between treatment methods and prognosis in the training data. Prognosis is information about the course of treatment given to the patient, such as the 5-year survival rate or whether or not there was a recurrence. Prognosis is assumed to be pre-assigned to the pathological tissue images in the training dataset. By training the GNN using the above-described training data, the GNN learning unit 104 can create a predictive model that takes images as input and can predict the relationship between the tissues contained in the images, linguistic descriptions of tissue morphology, case classes, treatment methods, and prognosis.
[0055] By using the above-described prediction model, the prediction unit 200 can output not only the optimal treatment but also the probability of survival and the likelihood of recurrence.
[0056] Furthermore, the items that can be added to the training data are not limited to prognosis. The GNN learning unit 104 can generate training data by adding items related to partial images, tissue morphology, case class, and treatment methods.
[0057] <Second Embodiment> Figure 10 is a block diagram showing the functional configuration of the tissue morphology representation device of the second embodiment. The tissue morphology representation device 300 comprises a first partial image generation means 301, a prediction means 302, a graph generation means 303, and a language output means 304.
[0058] Figure 11 is a flowchart of the processing performed by the tissue morphology representation device of the second embodiment. The first partial image generation means 301 generates a partial image smaller than the input image from an input image that includes the tissue morphology of the target patient (step S301). The prediction means 302 inputs the partial image to a trained GNN and predicts the relationship between the tissue morphology contained in the partial image and the linguistic representation of the tissue morphology, the relationship between the linguistic representation of the tissue morphology and the case class, and the relationship between the case class and the treatment method (step S302). The graph generation means 303 generates a prediction graph based on the prediction results (step S303). The language output means 304 inputs the prediction graph to a large-scale language model and outputs a response from the large-scale language model (step S304).
[0059] According to the tissue morphology representation device 300 of the second embodiment, it becomes possible to diagnose pathological tissue images using appropriate linguistic representation. Furthermore, the tissue morphology representation device 300 can support the user's decision-making.
[0060] Some or all of the above embodiments may also be described as follows, but are not limited to the following:
[0061] (Note 1) A tissue morphology representation device comprising: a first partial image generation means that generates a partial image smaller than the input image from an input image including the tissue morphology of a target patient; a prediction means that inputs the partial image to a trained GNN and predicts the relationship between the tissue morphology included in the partial image and the linguistic expression of the tissue morphology, the relationship between the linguistic expression of the tissue morphology and the case class, and the relationship between the case class and the treatment method; a graph generation means that generates a prediction graph based on the prediction results; and a language output means that inputs the prediction graph to a large-scale language model and outputs a response from the large-scale language model.
[0062] (Note 2) The tissue morphology representation device of Note 1, comprising a first image embedding means that converts the partial image into a feature vector and performs dimensionality reduction, wherein the prediction means uses the feature vector of the dimensionality-reduced partial image as input to a trained GNN.
[0063] (Note 3) The organizational structure representation device of Note 1, wherein the graph generation means converts the predicted graph into text using a data description language for describing the graph structure, and the language output means inputs the text as a prompt into the large-scale language model.
[0064] (Note 4) The language output means is the organizational morphology representation device of Note 1, which outputs the response from the large-scale language model and the prediction graph.
[0065] (Note 5) The response from the large-scale language model is the tissue morphology representation device of Note 1, which includes a suggestion for the optimal treatment for the input image and the reason for selecting the optimal treatment.
[0066] (Note 6) The tissue morphology representation device of Note 1, comprising a learning means for training a GNN, which is a machine learning model, wherein the learning means trains the GNN using graph data that associates partial images with linguistic representations of tissue morphology, linguistic representations of tissue morphology with case classes, and case classes with treatment methods as training data, and outputs the trained GNN.
[0067] (Note 7) The aforementioned graph data is the tissue morphology representation device of Note 6, which represents partial images, linguistic descriptions of tissue morphology, case classes, and treatment methods as nodes, and the relationships between nodes as links.
[0068] (Note 8) The tissue morphology representation device according to Note 6, comprising: a second partial image generation means for generating a partial image smaller than the pathological tissue image from a pathological tissue image included in a training dataset; a second image embedding means for converting a plurality of partial images generated by the second partial image generation means into feature vectors and mapping them to a feature space; a tissue classification means for dividing the feature space into a plurality of regions; and a learning data generation means for obtaining associations between each divided region and a linguistic expression of tissue morphology, and generating learning data based on the obtained associations between each region and the linguistic expression of tissue morphology, and the case class and treatment method associated with the pathological tissue image.
[0069] (Note 9) The tissue classification means is a tissue morphology representation device according to Note 8, wherein the tissue classification means selects a local peak of density of each point mapped to the feature space as a representative point, and divides the feature space into a plurality of regions based on the representative point.
[0070] (Note 10) The tissue classification means is a tissue morphology representation device according to Note 8 that generates learning data by associating a linguistic representation of the tissue morphology assigned to each of the regions with a partial image belonging to that region.
[0071] (Note 11) A method for representing tissue morphology, comprising: generating a partial image smaller than the input image from an input image including the tissue morphology of a target patient; inputting the partial image into a trained GNN; predicting the relationship between the tissue morphology contained in the partial image and the linguistic expression of the tissue morphology, the relationship between the linguistic expression of the tissue morphology and the case class, and the relationship between the case class and the treatment method; generating a prediction graph based on the prediction results; inputting the prediction graph into a large-scale language model; and outputting a response from the large-scale language model.
[0072] (Note 12) A recording medium that contains a program that causes a computer to execute the following processes: generating a smaller partial image from an input image including the tissue morphology of a target patient; inputting the partial image into a trained GNN; predicting the relationship between the tissue morphology contained in the partial image and the linguistic representation of the tissue morphology, the relationship between the linguistic representation of the tissue morphology and the case class, and the relationship between the case class and the treatment method; generating a prediction graph based on the prediction results; inputting the prediction graph into a large-scale language model; and outputting a response from the large-scale language model.
[0073] Furthermore, some or all of the configurations described in Appendices 2 to 10, which are subordinate to Appendice 1 above, may also be subordinate to Appendices 11 and 12 in the same way as those described in Appendices 2 to 10. Moreover, not limited to Appendices 1, 11, and 12, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.
[0074] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure.
[0075] 10 Organizational morphology representation device 100 Learning unit 101 Image segmentation unit 102 Image embedding unit 103 Organizational classification unit 104 GNN learning unit 200 Prediction unit 201 Image segmentation unit 202 Image embedding unit 203 Organizational languageization unit 204 LLM unit
Claims
1. A tissue morphology representation device comprising: a first partial image generation means that generates a partial image smaller than the input image from an input image including the tissue morphology of a target patient; a prediction means that inputs the partial image to a trained GNN and predicts the relationship between the tissue morphology contained in the partial image and the linguistic expression of the tissue morphology, the relationship between the linguistic expression of the tissue morphology and the case class, and the relationship between the case class and the treatment method; a graph generation means that generates a prediction graph based on the prediction results; and a language output means that inputs the prediction graph to a large-scale language model and outputs a response from the large-scale language model.
2. The tissue morphology representation apparatus according to claim 1, comprising a first image embedding means that converts the partial image into a feature vector and performs dimensionality reduction, wherein the prediction means uses the feature vector of the dimensionality-reduced partial image as input to a trained GNN.
3. The organizational structure representation device according to claim 1, wherein the graph generation means converts the predicted graph into text using a data description language for describing the graph structure, and the language output means inputs the text as a prompt into the large-scale language model.
4. The organizational morphology representation device according to claim 1, wherein the language output means outputs the response from the large-scale language model and the prediction graph.
5. The tissue morphology representation device of claim 1, wherein the response from the large-scale language model includes a suggestion for the optimal treatment for the input image and the reason for selecting the optimal treatment.
6. The tissue morphology representation device according to claim 1, comprising a learning means for training a GNN, which is a machine learning model, wherein the learning means trains the GNN using graph data that associates partial images with linguistic representations of tissue morphology, linguistic representations of tissue morphology with case classes, and case classes with treatment methods as training data, and outputs the trained GNN.
7. The tissue morphology representation device according to claim 6, wherein the graph data is graph data in which partial images, linguistic descriptions of tissue morphology, case classes, and treatment methods are represented by nodes, and the relationships between nodes are represented by links.
8. The tissue morphology representation device according to claim 6, comprising: a second partial image generation means for generating a partial image smaller than the pathological tissue image from a pathological tissue image included in a training dataset; a second image embedding means for converting a plurality of partial images generated by the second partial image generation means into feature vectors and mapping them to a feature space; a tissue classification means for dividing the feature space into a plurality of regions; and a learning data generation means for obtaining associations between each divided region and a linguistic expression of tissue morphology, and generating learning data based on the obtained associations between each region and the linguistic expression of tissue morphology, and the case class and treatment method associated with the pathological tissue image.
9. The tissue morphology representation apparatus according to claim 8, wherein the tissue classification means selects a local peak of density of each point mapped to the feature space as a representative point, and divides the feature space into a plurality of regions based on the representative point.
10. The tissue classification means generates training data by associating a linguistic representation of the tissue morphology assigned to each region with a partial image belonging to that region, according to claim 8.
11. A method for representing tissue morphology, comprising: generating a smaller partial image from an input image including the tissue morphology of a target patient; inputting the partial image into a trained GNN; predicting the relationship between the tissue morphology contained in the partial image and the linguistic representation of the tissue morphology, the relationship between the linguistic representation of the tissue morphology and the case class, and the relationship between the case class and the treatment method; generating a prediction graph based on the prediction results; inputting the prediction graph into a large-scale language model; and outputting a response from the large-scale language model.
12. A recording medium that contains a program that causes a computer to execute the following processes: generating a smaller partial image from an input image including the tissue morphology of a target patient; inputting the partial image into a trained GNN; predicting the relationship between the tissue morphology contained in the partial image and the linguistic representation of the tissue morphology, the relationship between the linguistic representation of the tissue morphology and the case class, and the relationship between the case class and the treatment method; generating a prediction graph based on the prediction results; inputting the prediction graph into a large-scale language model; and outputting a response from the large-scale language model.