Method and apparatus for outputting medical information

Abstract

A computing apparatus according to an aspect comprises: at least one memory storing at least one instruction; and at least one processor operating according to the at least one instruction, wherein the at least one processor may be configured to: identify at least one object in a pathology slide image by analyzing the pathology slide image using a machine learning model; register gene expression information corresponding to the at least one object, included in spatial transcriptomics information, with the at least one object represented in the pathology slide image; and control a display apparatus to output, on the basis of a result of the registration, information about at least one biomarker corresponding to the at least one object.

Description

Method and device for outputting medical information

[0001] The present disclosure relates to a method and apparatus for outputting medical information. More specifically, the present disclosure relates to a method and apparatus for providing medical information using pathology slide images and spatial transcriptomics information.

[0002] Recently, technologies are being developed to predict medical information about subjects by analyzing medical images through machine learning models. Representative examples include machine learning models that diagnose patient diseases (e.g., cancer) by analyzing medical images.

[0003] Pathological analysis of tissue samples is crucial for diagnosing diseases such as cancer and determining treatment methods. Traditionally, pathologists have relied on the visual examination of Hematoxylin and Eosin (H&E) stained pathology slides to analyze cellular histomorphology and determine the presence and grade of cancer. Recently, however, technologies have advanced that utilize artificial intelligence (AI), particularly machine learning models, to analyze pathology images, thereby enabling objective and consistent analysis at the cell or tissue level.

[0004] The present disclosure aims to accurately and efficiently register and integrate gene expression information obtained from morphological information obtained from pathology slide images and spatial transcriptome information at the single-cell level.

[0005] In addition, the present disclosure aims to automatically identify novel biomarkers for various cell types based on the results of the integrated analysis, to determine the quantitative relationship between cell morphological characteristics and gene expression, and to perform gene expression profiling considering the tissue context.

[0006] Ultimately, the present disclosure aims to provide a method and apparatus for providing clinically useful medical information, such as predicting a patient's responsiveness to treatment or discovering new therapeutic targets, through such analysis.

[0007] The technical challenges to be solved are not limited to those mentioned above, and other technical challenges may exist.

[0008] A computing device according to one aspect comprises: at least one memory in which at least one instruction is stored; and at least one processor that operates according to the at least one instruction. The at least one processor can identify at least one object in the pathology slide image by analyzing the pathology slide image using a machine learning model, register gene expression information corresponding to the at least one object included in spatial transcriptomics information with the at least one object represented in the pathology slide image, and control a display device to output information about at least one biomarker corresponding to the at least one object based on the result of the registration.

[0009] A method for outputting medical information according to another aspect may include: identifying at least one object in a pathology slide image by analyzing the pathology slide image using a machine learning model; registering gene expression information corresponding to the at least one object included in spatial transcriptomics information with the at least one object expressed in the pathology slide image; and outputting information about at least one biomarker corresponding to the at least one object based on the result of the registration.

[0010] A computer-readable recording medium according to another aspect may include a recording medium that records a program for executing the above-described method on a computer.

[0011] FIG. 1 is a diagram illustrating an example of a computing device outputting medical information according to one embodiment.

[0012] FIG. 2a is a configuration diagram illustrating an example of a user terminal according to one embodiment.

[0013] FIG. 2b is a configuration diagram illustrating an example of a server according to one embodiment.

[0014] FIG. 3 is a flowchart illustrating an example of a method for outputting medical information according to one embodiment.

[0015] FIG. 4 is a diagram illustrating an example of a processor identifying an object in a pathology slide image according to one embodiment.

[0016] FIG. 5 is a diagram illustrating an example in which a processor according to one embodiment matches gene expression information with an object represented on a pathology slide image.

[0017] FIG. 6 is a diagram illustrating an example in which information about at least one biomarker is output according to one embodiment.

[0018] FIG. 7 is a diagram illustrating another example in which information about at least one biomarker according to one embodiment is output.

[0019] FIG. 8 is a diagram illustrating another example in which information about at least one biomarker is output according to one embodiment.

[0020] FIG. 9 is a diagram illustrating another example in which information about at least one biomarker according to one embodiment is output.

[0021] FIG. 10 is a diagram illustrating an example in which information about at least one biomarker according to one embodiment is used to train a machine learning model.

[0022] FIG. 11 is a drawing for illustrating an example of a system for outputting medical information according to one embodiment.

[0023] A computing device according to one aspect comprises: at least one memory in which at least one instruction is stored; and at least one processor that operates according to the at least one instruction. The at least one processor can identify at least one object in the pathology slide image by analyzing the pathology slide image using a machine learning model, register gene expression information corresponding to the at least one object included in spatial transcriptomics information with the at least one object represented in the pathology slide image, and control a display device to output information about at least one biomarker corresponding to the at least one object based on the result of the registration.

[0024] The terms used in the embodiments have been selected to be as close as possible to currently widely used general terms; however, these may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant description section. Therefore, terms used in the specification must be defined not merely by their names, but based on their meanings and the content throughout the specification.

[0025] When a part of the specification is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "unit" and "module" as used in the specification refer to a unit that performs at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0026] Additionally, terms including ordinal numbers, such as "first" or "second," used in the specification may be used to describe various components, but said components shall not be limited by said terms. Such terms may be used for the purpose of distinguishing one component from another.

[0027] In the following, "medical information" may refer to any medically meaningful information or clinical information of a patient that can be extracted from medical images. Medical images may include not only pathology slide images but also radiographic images (X-ray, CT, MRI, PET, etc.). For example, medical information may include at least one of an immune phenotype, genotype, expression type, biomarker, tumor purity, information regarding RNA, tumor microenvironment, cancer regimen expressed in the pathology slide image, survival information, treatment response, treatment outcome, genetic characteristics, and medical records.

[0028] In addition, medical information may also include anatomical structural information extracted from medical images, types of lesions, locations and sizes of lesions, morphological features of lesions (e.g., boundaries, texture, density), functional indicators (e.g., blood flow, metabolic activity), abnormal findings of organs, indicators related to treatment prognosis obtained from medical images, information regarding findings obtained by analyzing medical images using artificial intelligence models, abnormality scores of said findings, reliability of said findings, and image biomarkers (radiomic features).

[0029] In addition, medical information may include findings such as the presence or absence of nodules in the medical image, signs of pneumonia, the presence of pneumothorax, the location and type of fractures, the location, size, shape, and boundary characteristics of masses, the distribution of microcalcifications, asymmetry, and breast tissue density and structural distortion. Such findings may be calculated along with, but are not limited to, an abnormality score or risk score for the relevant image.

[0030] Additionally, medical information may include, but is not limited to, the area, location, and size of specific tissues (e.g., cancer tissue, cancer stromal tissue, etc.) and / or specific cells (e.g., tumor cells, lymphocytes, macrophages, endothelial cells, fibroblasts, etc.) within the medical image, diagnostic information of cancer, information related to the patient's probability of developing cancer, and / or medical conclusions related to cancer treatment.

[0031] In addition, medical information may include not only quantified values ​​obtainable from medical images but also information visualizing the values, predictive information based on the values, image information, statistical information, etc. For example, medical information may be provided to a user terminal or output through a display device.

[0032] Embodiments are described in detail below with reference to the attached drawings. However, embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0033] FIG. 1 is a diagram illustrating an example of a computing device outputting medical information according to one embodiment.

[0034] Referring to FIG. 1, the computing device (30) can analyze the pathology slide image (10) and spatial transcriptome information (20) to output medical information (40).

[0035] Spatial transcriptome information (20) refers to information obtained through spatial transcriptome analysis. For example, spatial transcriptome information (20) may include information obtained through spatial transcriptome analysis at the single-cell level. Specifically, spatial transcriptome information (20) may include sequence data obtained through spatial transcriptome analysis, single-cell level gene expression information confirmed by data processing of the sequence data, etc.

[0036] Spatial transcriptome analysis is a molecular profiling method that enables the measurement of gene expression in tissue samples and the mapping of the locations where genes are expressed. The relative positional relationship between cells and tissues is crucial for understanding the normal development of cells or tissues and the pathology of diseases. However, because conventional Bulk-RNAseq analyzes a mixture of various tissues and cells simultaneously, it is impossible to determine detailed patterns of gene expression within a spatial context. Spatial transcriptome analysis allows for the identification of gene expression patterns within a spatial context. Consequently, this can improve not only the understanding of diseases but also the accuracy of diagnosis and treatment.

[0037] The spatial transcriptome information (20) includes genetic information corresponding to the pathology slide image (10) and / or at least one grid included in the pathology slide image. For example, the pathology slide image (10) may be divided into multiple grids, and a single grid may be an area of ​​1 mm * 1 mm, but is not limited thereto.

[0038] The computing device (30) can analyze the pathology slide image (10) using a machine learning model and align the pathology slide image (10) with spatial transcriptome information (20). Accordingly, the computing device (30) can generate medical information (40).

[0039] A machine learning model refers to a statistical learning algorithm implemented based on the structure of a biological neural network, or a structure that executes such an algorithm. For example, a machine learning model may represent a model capable of problem-solving, in which nodes—artificial neurons that form a network through synaptic connections as in biological neural networks—learn by repeatedly adjusting the weights of the synapses to reduce the error between the correct output corresponding to a specific input and the inferred output. For example, a machine learning model may include arbitrary probability models, neural network models, etc., used in artificial intelligence learning methods such as deep learning.

[0040] For example, a machine learning model can be implemented as a multilayer perceptron (MLP) composed of multiple layers of nodes and connections between them. The machine learning model according to the present embodiment can be implemented using one of various artificial neural network model structures including an MLP. For example, the machine learning model may be composed of an input layer that receives an input signal or data from the outside, an output layer that outputs an output signal or data corresponding to the input data, and at least one hidden layer located between the input layer and the output layer, which receives a signal from the input layer, extracts a feature, and transmits it to the output layer. The output layer receives a signal or data from the hidden layer and outputs it to the outside.

[0041] Accordingly, the machine learning model can be trained to extract information about one or more objects (e.g., cells, tissues, structures, etc.) included in the pathology slide image (10).

[0042] Meanwhile, gene expression information is of great importance in understanding biological samples. In particular, spatial transcriptome analysis enables the measurement of gene expression while maintaining the spatial location of cells within tissue samples, thereby allowing for the identification of the genetic characteristics of cells at specific locations.

[0043] As such, morphological information obtained through the analysis of pathology slide images (10) and spatial transcriptome information (20) obtained through spatial transcriptome analysis are essential information for deeply understanding cancer samples. However, since the two types of data described above are generated based on different technologies, there is a technical difficulty in that the coordinate systems between the two types of data do not match. Therefore, there is a need for the development of a technology that can accurately link and analyze morphological features and gene expression information at the single-cell level.

[0044] A computing device (30) according to one embodiment identifies objects such as cells or tissues in a pathology slide image (10) using a machine learning model and aligns gene expression information included in spatial transcriptome information (20) with the objects. Here, the alignment is performed based on the coordinates of cells detected from different images to increase accuracy.

[0045] The computing device (30) generates and outputs biomarker information based on the matching results, including marker genes by cell type, subtype information based on cell shape and gene expression, gene expression profile based on tissue location, or clinical information such as the patient's treatment responsiveness. Through this, the computing device (30) can output medical information (40) based on accurate and multifaceted analysis.

[0046] Hereinafter, with reference to FIGS. 2a to 11, examples are described in which a computing device (30) aligns and analyzes a pathology slide image (10) and spatial transcriptome information (20) and generates medical information (40).

[0047] For example, the computing device (20) may be a user terminal or a server. In other words, the operations performed by the computing device (20) may be performed by a user terminal or a server. Alternatively, some of the operations performed by the computing device (20) may be performed by a user terminal, and the remainder may be performed by a server.

[0048] A user terminal may be an electronic device comprising a display device and a device for receiving user input (e.g., a keyboard, a mouse, etc.), and including memory and a processor. Additionally, the display device may be implemented as a touch screen to perform the function of receiving user input. For example, the user terminal may include, but is not limited to, notebook PCs, desktop PCs, laptops, tablet computers, smartphones, etc.

[0049] A server may be a device that communicates with external devices (e.g., user terminals). For example, a server may be a device that stores various data, including medical information and information about machine learning models. Alternatively, a server may be an electronic device that includes memory and a processor and possesses its own computing capabilities. For example, a server may be a cloud server or an on-premise server.

[0050] Hereinafter, examples of a user terminal and a server will be described with reference to FIGS. 2a and 2b.

[0051] FIG. 2a is a configuration diagram illustrating an example of a user terminal according to one embodiment.

[0052] Referring to FIG. 2a, the user terminal (100) includes a processor (110), memory (120), an input / output interface (130), and a communication module (140). For convenience of explanation, FIG. 2a only illustrates components related to the present invention. Accordingly, other general-purpose components may be included in the user terminal (100) in addition to the components illustrated in FIG. 2a. Furthermore, it is obvious to those skilled in the art that the processor (110), memory (120), input / output interface (130), and communication module (140) illustrated in FIG. 2a may be implemented as independent devices.

[0053] The processor (110) can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Here, instructions may be provided from memory (120) or an external device (e.g., a server (200), etc.). Additionally, the processor (110) can control the overall operation of other components included in the user terminal (100).

[0054] The processor (110) identifies at least one object in the pathology slide image (10) by analyzing the pathology slide image (10) using a machine learning model.

[0055] Here, at least one object may include at least one of the type of at least one cell, the shape of at least one cell, the shape of each of the components included in at least one cell (e.g., cell membrane, cytoplasm, cell nucleus) expressed on the pathology slide image (10), or at least one of the type of at least one tissue expressed on the pathology slide image (10).

[0056] An example of the processor (110) analyzing a pathology slide image (10) to identify an object is described later with reference to step 310 of FIG. 3.

[0057] The processor (110) matches gene expression information corresponding to at least one object included in the spatial transcriptome information (20) with at least one object represented in the pathology slide image (10).

[0058] For example, the processor (110) can match gene expression information with at least one object by matching a medical image (e.g., a fluorescence image) in which spatial transcriptome information (20) is expressed with a pathology slide image (10).

[0059] As an example, the processor (110) can match the medical image and the pathology slide image (10) based on at least one feature extracted from each of the pathology slide image (10) and the medical image. As another example, the processor (110) can match the medical image and the pathology slide image (10) based on a first coordinate corresponding to at least one object represented on the pathology slide image (10) and a second coordinate corresponding to at least one object represented on the medical image.

[0060] An example of the processor (110) matching gene expression information with an object represented in a pathology slide image (10) is described later with reference to step 320 of FIG. 3.

[0061] The processor (110) outputs information about at least one biomarker corresponding to at least one object based on the result of the match.

[0062] Here, biomarkers may include all of the following: cell type, cell characteristics, and biomarkers that predict therapeutic responsiveness. For example, biomarkers may include genes and / or products of specific genes (e.g., RNA and proteins).

[0063] As an example, the processor (110) can output at least one of information about each of the plurality of cells expressed in the pathology slide image (10), information about at least one gene expressed in a cell specified by user input among the plurality of cells, or information about at least one gene expressed by type of the plurality of cells.

[0064] As another example, the processor (110) can output multiple cells represented in the pathology slide image (10) so that the multiple cells are distinguished from each other by subtypes in which the multiple cells are classified.

[0065] As another example, the processor (110) can output at least one of information about each of the plurality of tissues expressed in the pathology slide image (10), information about at least one gene expressed in the tissue specified by user input among the plurality of tissues, or information about at least one gene expressed by type of the plurality of tissues.

[0066] As another example, the processor (110) can output information corresponding to subtypes of multiple cells expressed in the pathology slide image (10).

[0067] For example, subtypes may be determined based on cell morphology and / or gene expression. Additionally, a single cell type may include at least one subtype. For example, subtypes of multiple cells may be determined based on cell morphology and / or gene expression using an unsupervised clustering method or a supervised clustering method, but are not limited to the examples described above. Subtypes based on cell morphology may be generated by considering at least one of the cell shape, the size of the nucleus, the shape of the nucleus, or the shape of the cytoplasm. Additionally, subtypes based on gene expression may be generated based on the expression of genes associated with aggression (e.g., less aggressive or more aggressive).

[0068] An example of the processor (110) outputting information about a biomarker is described later with reference to step 330 of FIG. 3.

[0069] The processor (110) may be implemented as an array of multiple logic gates, or as a combination of a general-purpose microprocessor and memory storing a program that can be executed on the microprocessor. For example, the processor (110) may include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor (110) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor (110) may refer to a combination of processing devices such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a digital signal processor (DSP) core, or any other combination of such configurations.

[0070] The memory (120) may include any non-transient computer-readable recording medium. As an example, the memory (120) may include a permanent mass storage device such as a random access memory (RAM), read-only memory (ROM), disk drive, solid state drive (SSD), or flash memory. As another example, a permanent mass storage device such as a ROM, SSD, flash memory, or disk drive may be a separate permanent storage device distinct from the memory. Additionally, the memory (120) may store an operating system (OS) and at least one program code (e.g., code for the processor (110) to perform an operation described later with reference to FIGS. 3 through 11).

[0071] These software components may be loaded from a computer-readable recording medium separate from the memory (120). This separate computer-readable recording medium may be a recording medium that can be directly connected to a user terminal (100), and may include, for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, memory card, etc. Alternatively, the software components may be loaded into the memory (120) through a communication module (140) that is not a computer-readable recording medium. For example, at least one program may be loaded into the memory (120) based on a computer program (e.g., a computer program for a processor (110) to perform an operation described later with reference to FIGS. 3 to 11) which is installed by files provided through the communication module (140) by developers or a file distribution system that distributes installation files for an application.

[0072] The input / output interface (130) may be a means for interfacing with a device for input or output (e.g., keyboard, mouse, etc.) that may be connected to or included in the user terminal (100). In FIG. 2a, the input / output interface (130) is shown as an element configured separately from the processor (110), but is not limited thereto, and the input / output interface (130) may be configured to be included in the processor (110).

[0073] The communication module (140) may provide a configuration or function for the server (200) and the user terminal (100) to communicate with each other via a network. Additionally, the communication module (140) may provide a configuration or function for the user terminal (100) to communicate with other external devices. For example, control signals, commands, data, etc. provided under the control of the processor (110) may be transmitted to the server (200) and / or external devices via the communication module (140) and the network.

[0074] Meanwhile, although not illustrated in FIG. 2a, the user terminal (100) may further include a display device. Alternatively, the user terminal (100) may be connected to an independent display device via wired or wireless communication to transmit and receive data to and from each other. For example, medical images, medical information, information related to machine learning models, etc., may be provided to the user through the display device.

[0075] FIG. 2b is a configuration diagram illustrating an example of a server according to one embodiment.

[0076] Referring to FIG. 2b, the server (200) includes a processor (210), memory (220), and a communication module (230). For convenience of explanation, FIG. 2b shows only the components related to the present invention. Accordingly, other general-purpose components may be included in the server (200) in addition to the components shown in FIG. 2b. Furthermore, it is obvious to those skilled in the art that the processor (210), memory (220), and communication module (230) shown in FIG. 2b may be implemented as independent devices.

[0077] The processor (210) may receive data for performance evaluation from at least one of the memory (220), the user terminal (100), and other external devices. The processor (210) may use a machine learning model to identify objects in the pathology slide image (10), match gene expression information corresponding to objects included in spatial transcriptome information with objects represented in the pathology slide image (10), or output information about biomarkers based on the result of the match. Alternatively, the processor (210) may transmit information about biomarkers to the user terminal (100).

[0078] In other words, at least one of the operations of the processor (110) described above with reference to FIG. 2a can be performed by the processor (210). In this case, the user terminal (100) can output information transmitted from the server (200) through a display device.

[0079] Meanwhile, since the implementation example of the processor (210) is the same as the implementation example of the processor (110) described above with reference to FIG. 2a, a detailed description is omitted.

[0080] Various data, such as data generated according to the operation of the processor (210), can be stored in the memory (220). Additionally, an operating system (OS) and at least one program (e.g., a program required for the processor (210) to operate) can be stored in the memory (220).

[0081] Meanwhile, since the implementation example of the memory (220) is the same as the implementation example of the memory (120) described above with reference to FIG. 2a, a detailed description is omitted.

[0082] The communication module (230) may provide a configuration or function for the server (200) and the user terminal (100) to communicate with each other via a network. Additionally, the communication module (230) may provide a configuration or function for the server (200) to communicate with other external devices. For example, control signals, commands, data, etc. provided under the control of the processor (210) may be transmitted to the user terminal (100) and / or external devices via the communication module (230) and the network.

[0083] FIG. 3 is a flowchart illustrating an example of a method for outputting medical information according to one embodiment.

[0084] The method illustrated in FIG. 3 consists of steps processed chronologically in the computing device (30, 100, 200) or processor (110, 210) illustrated in FIG. 1 to 2b. Therefore, even if details are omitted below, the details described above regarding the computing device (30, 100, 200) or processor (110, 210) may also be applied to the method illustrated in FIG. 3. Additionally, as described above with reference to FIG. 2b, at least one of the steps performed by the processor (110) below may be processed by the processor (210).

[0085] In step 310, the processor (110) identifies at least one object in the pathology slide image by analyzing the pathology slide image using a machine learning model.

[0086] For example, the processor (110) can use a machine learning model to analyze pathology slide images and classify cells and / or tissues.

[0087] A pathology slide image refers to an image obtained by digitizing a stained tissue slide using a scanner after staining tissue collected from the human body using a specific method. For example, a pathology slide image may be an H&E image stained with hematoxylin and eosin, but is not limited thereto. For another example, a pathology slide image may be an immunohistochemistry (IHC)-stained image. For example, immunohistochemistry staining may be a method of visualizing antibodies conjugated with peroxidase via a DAB (3,3'-diaminobendidin) reaction, but is not limited thereto. In other words, the generation of a pathology slide image is not limited to the staining type of a specific antibody. For another example, a pathology slide image may include a fluorescence image obtained through immunofluorescence staining. For example, the fluorescent image may include a signal visualizing a cell nucleus stained with DAPI (4′,6-diamidino-2-phenylindole) and a ribosome-rich cytoplasmic region using a fluorescent probe complementary to 18S ribosomal RNA (18S rRNA).

[0088] As another example, a synthetic pathology slide image may be used as the pathology slide image. The processor (110) may generate a synthetic pathology slide image stained in a second method by converting at least one of the color, brightness, or style of the slide image stained in a first method. For example, the processor (110) may generate a synthetic H&E image by converting the color information of a fluorescent image stained with cell nuclei and cytoplasm using DAPI, etc. If there is no slide stained in a staining method that allows the user to verify the characteristics of the object to be verified, or if the number of slides is insufficient, the processor (110) may generate a synthetic pathology slide image stained in a method specified by the user, identify at least one object in the said synthetic slide image, and perform an embodiment according to the present disclosure.

[0089] For example, a machine learning model may refer to a statistical learning algorithm implemented based on the structure of a biological neural network, or a structure that executes such an algorithm.

[0090] The pathology slide image may be a whole slide image or any one of the patches into which the whole slide image is divided. For example, the processor (110) may analyze the whole slide image to classify cells and / or tissues. Or, the processor (110) may analyze the patches to classify cells and / or tissues. Hereinafter, the pathology slide image may refer to the whole slide image or the patches.

[0091] Hereinafter, with reference to FIG. 4, an example of a processor (110) identifying at least one object in a pathology slide image is described.

[0092] FIG. 4 is a diagram illustrating an example of a processor identifying an object in a pathology slide image according to one embodiment.

[0093] Referring to FIG. 4, the processor (110) can identify at least one object (420) by analyzing a pathology slide image (410).

[0094] Here, the object (420) may include at least one of the type of cell, the shape of the cell, the shape of each of the components included in the cell, or the type of tissue expressed on the pathology slide image (410).

[0095] For example, the identified object (420) may include a cell-level object and a tissue-level object. The cell-level object may include the type of cell (e.g., tumor cell, lymphocyte, fibroblast, endothelial cell, macrophage, etc.), the shape of the cell (e.g., size of the cell, size of the nucleus, irregularity of the shape of the nucleus, etc.), and the shape of the cell components (e.g., shape of the cell membrane, cytoplasm, nucleus). Additionally, the tissue-level object may include the type of tissue (e.g., cancer area, cancer stroma area, necrosis area, tertiary lymphoid structure (TLS), etc.).

[0096] The processor (110) can analyze a pathology slide image (410) or patches into which the pathology slide image (410) is divided using a machine learning model. For example, the processor (110) can generate patches by dividing the pathology slide image (410) into a predetermined size (e.g., 1216*1216 pixels). Then, the processor (110) can analyze the patches.

[0097] Accordingly, the processor (110) can detect and classify cells contained in the pathology slide image (410) through a machine learning model. Additionally, the processor (110) can detect and classify tissues contained in the pathology slide image (410) through a machine learning model.

[0098] For example, the processor (110) can output detection results in the form of layers representing tissues on the pathology slide image (410) using a machine learning model. In this case, the machine learning model can be trained to detect regions within the pathology slide image (410) that correspond to tissues within the reference pathology slide images by using training data that includes a plurality of reference pathology slide images (or patches) and a plurality of reference label information.

[0099] And, the processor (110) can perform classification on multiple tissues expressed in the pathology slide image (410). For example, the processor (110) can classify the tissues of the pathology slide image (410) into a cancer area or other area. Or, the processor (110) can classify the tissues of the patch into any one of a cancer area, a cancer stroma area, a necrotic area, a tertiary lymphocyte structure area, and a background area.

[0100] However, the examples of the processor (110) classifying regions included in the pathology slide image (410) are not limited to those described above. In other words, without being limited to the regions described above (cancer region, cancer stroma region, necrosis region, tertiary lymphocyte structure and background region), the processor (110) can classify regions included in the pathology slide image (410) into multiple categories according to various criteria. For example, regions included in the pathology slide image (410) can be classified into multiple categories according to a pre-set criterion or a criterion set by the user.

[0101] And, the processor (110) can analyze the pathology slide image (410) to perform detection and classification of multiple cells.

[0102] First, the processor (110) can analyze the pathology slide image (410), detect cells from the pathology slide image (410), and output the detection results in the form of layers representing the cells.

[0103] The processor (110) can output detection results in the form of layers representing cells on a pathology slide image (410) using a machine learning model. In this case, the machine learning model can be trained to detect the location and type of cells within the reference pathology slide images (410) using training data that includes a plurality of reference pathology slide images (or patches) and a plurality of reference label information.

[0104] And, the processor (110) can perform classification on a plurality of cells included in the pathology slide image (410). For example, the processor (110) can classify the plurality of cells as tumor cells or other cells. Or, the processor (110) can classify the plurality of cells as at least one of tumor cells, lymphocyte cells, and other cells.

[0105] However, the examples of classifying cells expressed in the pathology slide image (410) by the processor (110) are not limited to those described above. In other words, without being limited to the cells described above (i.e., tumor cells, lymphocyte cells, and other cells), the processor (110) may classify the cells expressed in the pathology slide image (410) into multiple categories according to various criteria. The cells in the pathology slide image (410) may be grouped into multiple categories according to a preset criterion or a criterion set by the user.

[0106] Additionally, the processor (110) can use a machine learning model to analyze the pathology slide image (410) and identify the components contained in the cell. For example, the components may include a cell membrane, cytoplasm, and a cell nucleus.

[0107] As components are identified in the pathology slide image (410), the processor (110) can output information such as the location and shape of the components.

[0108] For example, the machine learning model may be a deep convolution network. Additionally, the machine learning model may be trained using fully supervised learning with reference pathology slide images (or patches) containing manual annotations as training data. However, the training method of the machine learning model is not limited to what has been described above. For example, the machine learning model may be trained using self-supervised learning with reference pathology slide images that do not contain annotations as training data.

[0109] Referring again to FIG. 3, in step 320, the processor (110) matches gene expression information corresponding to at least one object included in spatial transcriptome information with at least one object represented in the pathology slide image.

[0110] For example, the processor (110) can match gene expression information with at least one object by matching a medical image in which spatial transcriptome information is expressed with a pathology slide image. Here, the medical image may be a fluorescence image in which spatial transcriptome information is expressed.

[0111] Hereinafter, with reference to FIG. 5, an example is described in which a processor (110) matches gene expression information with an object represented in a pathology slide image.

[0112] FIG. 5 is a diagram illustrating an example in which a processor according to one embodiment matches gene expression information with an object represented on a pathology slide image.

[0113] Referring to FIG. 5, the processor (110) can match the morphological information of pathology slide images (510) in different coordinate systems with the gene expression information of spatial transcriptome information at the single-cell level. The medical image (520) illustrated in FIG. 5 may be a fluorescence image in which spatial transcriptome information is expressed.

[0114] As an example, the processor (110) can match the pathology slide image (510) and the medical image (520) based on at least one feature extracted from each of the pathology slide image (510) and the medical image (520). For example, the feature may be directly marked by a pathologist as a common feature of the images (510, 520) or may be automatically identified through an image analysis algorithm. The processor (110) can match the images (510, 520) by performing image transformation so that the features overlap well.

[0115] As another example, the processor (110) can match images (510, 520) based on a first coordinate corresponding to an object represented on a pathology slide image (510) and a second coordinate corresponding to an object represented on a medical image (520).

[0116] In other words, the processor (110) can perform matching of images (510, 520) based on cell coordinates detected by artificial intelligence, instead of directly matching the images (510, 520). Since the location of the cell is relatively less affected by the modality of the image, it can be matched more robustly and accurately.

[0117] First, the processor (110) can obtain the coordinates of cells in a pathology slide image (510) through a machine learning model. Then, the processor (110) can also obtain the coordinates of cells in a medical image (520) through a machine learning model.

[0118] And, the processor (110) can calculate an optimal transformation matrix for transforming one coordinate system into another coordinate system. For example, the calculation of the transformation matrix can be performed in a semi-automatic manner in which a pathologist specifies a reference point, or in an automatic manner in which the Hamming distance between binary images encoding the distribution of two sets of coordinates is repeatedly minimized.

[0119] When the conversion is complete, the processor (110) can calculate the physical distance between each cell coordinate in the registered coordinate space and match the closest pair (530). For example, through the matching (530), specific cells on the pathology slide image (510) and the gene expression information of the corresponding cells can be accurately linked with high reliability of an average of 2 µm or less.

[0120] However, iterative Hamming distance minimization is merely one example of matching images (510, 520). For example, the processor (110) can perform matching of images (510, 520) using different techniques depending on the distribution of cells, the size of tissues, the shape of tissues, etc. For example, the processor (110) can perform matching of images (510, 520) using an integer matrix that encodes the type of cells, rather than a binary matrix that encodes the presence or absence of cells.

[0121] Referring again to FIG. 3, in step 330, the processor (110) outputs information about at least one biomarker corresponding to at least one object based on the result of the match.

[0122] Here, biomarkers may include all of the following: cell type, cell characteristics, and biomarkers that predict therapeutic responsiveness. For example, biomarkers may include genes and / or products of specific genes (e.g., RNA and proteins), Tumor-Infiltrating Lymphocytes (TILs), Immune Phenotypes, and tissue-level characteristics (e.g., cancer tissue regions, cancer stroma regions).

[0123] In the present disclosure, biomarkers may be used as therapeutic targets or to predict a patient's response to treatment.

[0124] As an example, biomarkers can be used as therapy targets.

[0125] The processor (110) can generate and output relevant information necessary for a user to determine a treatment target. For example, the processor (110) can output information about genes that are highly expressed or less expressed in each patient with a good treatment outcome and a patient with a poor treatment outcome. Alternatively, the processor (110) can output information about cell subtypes of patients with good treatment outcomes and / or patients with poor treatment outcomes, and information about genes that are highly expressed or less expressed in those subtypes.

[0126] Additionally, the processor (110) can determine a biomarker as a treatment target and output it. For example, the processor (110) can output information about a gene recommended as a treatment target.

[0127] As another example, biomarkers can be used to predict a patient's responsiveness to treatment. Here, treatment responsiveness can be calculated using various indicators (e.g., grade, score, ratio, life expectancy, etc.) that demonstrate the degree of the patient's response to treatment.

[0128] For example, if a specific gene can be used as a biomarker, the processor (110) can analyze the patient's pathology slide images and predict that the treatment response will be good if it is determined that the biomarker is highly expressed. Here, the expression information of the biomarker may include the number or ratio of cells in which the expression rate of the biomarker (e.g., gene) among a specific type of cell (e.g., tumor cells) is greater than a certain ratio, and comparative information on the expression rate of the biomarker expressed in tumor cells and the expression rate of the biomarker expressed in immune cells.

[0129] Additionally, if a specific gene can be used as a biomarker, the processor (110) can analyze the patient's pathology slide images and predict that the patient's treatment responsiveness will be good if a cell subtype capable of confirming the expression of the corresponding gene is found. Additionally, the processor (110) can determine a numerical value capable of indicating the patient's treatment responsiveness by considering the ranking and ratio (the ratio of a specific subtype among the cell subtypes detected for the corresponding cell type) of the cell subtypes capable of confirming the expression of the biomarker.

[0130] In addition, the processor (110) can predict the treatment responsiveness of the patient by considering the expression information (e.g., expression rate, expression ratio, etc.) of the biomarker expressed in the tissue (e.g., cancer area, cancer stromal area, tertiary lymphocyte structure, etc.).

[0131] Hereinafter, examples of outputting information about at least one biomarker are described with reference to FIGS. 6 to 9. In the following, the processor (110) outputting information includes the operation of controlling a display device so that the information is output.

[0132] FIG. 6 is a diagram illustrating an example in which information about at least one biomarker is output according to one embodiment.

[0133] FIG. 6 illustrates a screen (600) on which information about a biomarker is displayed. For example, the processor (110) may display at least one of information about each of a plurality of cells represented in a pathology slide image, information about at least one gene expressed in a cell specified by user input among the plurality of cells, or information about at least one gene expressed by each type of the plurality of cells.

[0134] For example, when a user places a cursor over a specific cell (610) on a pathology slide image on a screen (600), information (620) regarding at least one gene expressed in the cell (610) (e.g., number of expressions per gene, whether the gene is expressed, etc.) may be overlaid on the pathology slide image and displayed. Here, the number of expressions per gene refers to information displayed such as gene A being xxx and gene B being yyy, and the expression of the gene refers to information displayed such as gene A being 'expressed' and gene B being 'not expressed'.

[0135] Additionally, information regarding each of the multiple cells depicted in the pathology slide image may be displayed on the screen (600). For example, the cells in the pathology slide image may be distinguished and displayed in different colors according to their type. Additionally, information (630) regarding which type the corresponding color represents may be displayed on the screen (600).

[0136] Additionally, information (640) regarding at least one gene expressed for each type of multiple cells may be displayed on the screen (600). The information (640) may include how much specific genes are expressed (e.g., expression rate) within a population of cells of a specific type (e.g., tumor cells, lymphocyte cells, etc.) for each cell type. For example, expression rates of EPCAM, MALL, CYP2B6, etc. for tumor cells, and expression rates of TRAC, CXCR4, PTPRC, etc. for lymphocyte cells may be displayed, respectively. Here, the expression rate may be calculated by dividing the total number of cells of a specific type by the number of cells of a specific type in which the gene is expressed. Through this, marker genes representing the characteristics of each cell type can be automatically identified.

[0137] Here, marker genes are genes that are characteristically expressed in specific types of cells (e.g., tumor cells, lymphocyte cells, etc.) and can be used to locate or identify specific cells in a patient's sample.

[0138] For example, the processor (110) can incorporate gene expression information based on the coordinates of the cell as described above with reference to step 320 of FIG. 3. Accordingly, the processor (110) can generate a gene expression profile for each of the cell types.

[0139] Additionally, the processor (110) can identify marker genes by performing differential gene expression analysis. For example, the processor (110) can determine the expression rates of multiple genes expressed in a specific cell and determine i) the gene with the highest expression rate as a marker gene, ii) at least one gene exceeding a predetermined expression rate standard as a marker gene, iii) determine a predetermined number of genes in order of the highest expression rate value as marker genes, or iv) determine a gene that is specifically highly expressed in the cell as a marker gene. The method of identifying marker genes is not limited to the examples described above.

[0140] FIG. 7 is a diagram illustrating another example in which information about at least one biomarker according to one embodiment is output.

[0141] FIG. 7 illustrates a screen (700) on which information about biomarkers is displayed. For example, the processor (110) can output multiple cells represented in a pathology slide image so that the multiple cells are distinguished from each other by subtypes in which the multiple cells are classified.

[0142] For example, information regarding each of the multiple cells represented in the pathology slide image may be displayed on the screen (700). For example, the cells in the pathology slide image may be distinguished and displayed in different colors according to their type. Additionally, information (710) regarding which type the corresponding color represents may be displayed on the screen (700).

[0143] Additionally, on the screen (700), multiple cells can be displayed separately from each other according to classified subtypes (721, 722, 723). The processor (110) can combine the morphological characteristics of the cells and gene expression information to classify the cells into more detailed subtypes and output related information.

[0144] For example, the processor (110) can divide cells into subtypes according to classification criteria selected by the user (e.g., size of the cell nucleus, expression level of a specific gene, etc.). Specifically, the user can select one or more categories to be analyzed using a menu (730) displayed on the screen (700). For example, categories can be set in various ways, such as cell size, size of the cell nucleus, staining pattern, etc.

[0145] Meanwhile, the user may also make additional settings for the category to be analyzed. Additionally, the user may adjust the number of subtypes by setting a threshold for cell size. Specifically, the user may set the number of subtypes to two by setting one threshold, or set the number of subtypes to three by setting two thresholds.

[0146] And, the processor (110) can distinguish and display cells belonging to each subtype on the pathology slide image using different colors or patterns (721, 722, 723). For example, the processor (110) can display the subtypes so that they are distinguished from each other by different colors, different patterns, different icons, different transparency, etc.

[0147] Additionally, the processor (110) can output gene expression information (750) for each gene according to each subtype to the screen (700). For example, the gene expression information may include the number or ratio of cells in which the gene is expressed among all cells classified as the corresponding subtype. Additionally, the gene expression information may include the number or ratio of cells in which the gene is expressed along with those classified as the corresponding subtype. Additionally, the gene expression information may include an importance score for each gene for the corresponding subtype. Here, the importance score refers to a numerical value indicating which gene is important. For example, assuming that subtype A is a small-sized cell, the importance score may be a score indicating how important (or large) the gene is to the characteristic of the cell being small.

[0148] Accordingly, the user can easily check how the regions within the pathology slide image are divided into subtypes through the information displayed on the screen (700). In addition, the user can also visually check how the subtypes are distributed within the pathology slide image.

[0149] FIG. 8 is a diagram illustrating another example in which information about at least one biomarker is output according to one embodiment.

[0150] FIG. 8 illustrates a screen (800) on which information about biomarkers is displayed. For example, the processor (110) may display at least one of information about each of a plurality of tissues represented in a pathology slide image, information about at least one gene expressed in a tissue specified by user input among the plurality of tissues, or information about at least one gene expressed by type of the plurality of tissues.

[0151] The function of a cell may vary depending on the environment of the surrounding tissue. A processor (110) can perform tissue context-dependent gene expression profiling using tissue segmentation information. Specifically, the processor (110) can subdivide specific types of cells (e.g., lymphocytes) according to the tissue in which the cells are located (e.g., cancer region, cancer stromal region, TLS, etc.). And, the processor (110) can analyze the gene expression information of these tissue-context specific cells.

[0152] As an example, the processor (110) can classify lymphocyte cells as sTIL (stromal-TIL) if they are located in the cancer stromal region, and as iTIL (intratumoral-TIL) if they are located in the cancer region. Through the above classification, the user can determine which gene is expressed at a high (or low) rate in each region (i.e., the cancer stromal region or the cancer region).

[0153] Accordingly, the gene expression characteristics of immune cells that directly attack cancer cells and immune cells that do not can be identified. Here, the gene expression characteristics may include i) the expression amount of a specific gene (i.e., how many RNAs are detected per cell), ii) how many cells the specific gene is expressed in (e.g., in what percentage of cells one or more RNAs are detected), and iii) how the statistics of i) and ii) described above show differences between iTIL and sTIL, or which gene shows the largest difference.

[0154] In addition, new cancer treatment targets can be identified by confirming the spatial gene expression patterns of immune-resistant cancer tissues.

[0155] As another example, the processor (110) can analyze the gene expression patterns of lymphocyte cells located within the TLS. That is, the processor (110) can detect the TLS in the pathology slide image and identify the spatial gene expression pattern of the TLS. Alternatively, the processor (110) can identify what types of cells (e.g., B-cells, T-cells) exist in the TLS and where each cell is located.

[0156] Referring to FIG. 8, immune phenotype information (810) based on sTIL and iTIL values ​​can be displayed on the screen (800). For example, an inflamed area, an immune-excluded area, and an immune-desert area can be distinguished from each other and displayed on the pathology slide image.

[0157] Additionally, tissue-specific segmentation information (820) of the pathology slide image may be displayed on the screen (800). For example, the processor (110) can divide and classify tissue regions in the pathology slide image through tissue segmentation. Then, the processor (110) can display each tissue (e.g., cancer region, cancer stromal region, necrotic region, TLS, etc.) separately on the pathology slide image.

[0158] Additionally, information regarding each of the multiple cells depicted in the pathology slide image may be displayed on the screen (800). For example, the cells in the pathology slide image may be distinguished and displayed in different colors according to their type. Additionally, information (830) regarding which type the corresponding color represents may be displayed on the screen (800).

[0159] Additionally, detailed information (840) regarding a specific area may be displayed on the screen (800). For example, the specific area may include an area of ​​interest to the user (e.g., an area corresponding to where the cursor is placed by the user, an area within a pre-set shape (850), etc.) or an area corresponding to each type of tissue (i.e., cancer area, cancer stromal area, necrotic area, TLS, etc.). For example, when the user places the cursor over a specific area, the amount of gene expression in the area may be displayed in a separate window (840).

[0160] Additionally, additional information (860) regarding a specific region may be further displayed on the screen (800). For example, the additional information (860) may include: i) information about what immune phenotype the region has; ii) information about the TIL density of the region (e.g., iTIL density, sTIL density); iii) the expression amount of each gene expressed in the region (e.g., number of gene A, number of gene B, etc.).

[0161] Additionally, the additional information (860) may include numerical information (e.g., density information, number information) of each gene expressed in each region (cancer region, cancer stromal region, necrotic region, TLS, etc.). For example, gene A within the cancer region may be output as xxxxx transcripts / mm2, etc. Additionally, the additional information (860) may include the gene expression amount or gene expression distribution (e.g., mean) of the cell spatial subtype (e.g., iTIL, sTIL, etc.) for each gene.

[0162] Additionally, the additional information (860) may include ranking information of the genes expressed in each region based on the degree of gene expression and / or numerical information of each expressed gene (e.g., density, number, etc.). For example, ranking information of the most expressed genes in iTILs includes '1) CD8A (x1 transcripts / mm 2 ), 2) GZMA (x2 transcripts / mm 2 ), 3) GZMK (x3 transcripts / mm 2 It can be displayed as )'.

[0163] Additionally, the additional information (860) may include the score of the Inflamed area, the score of the Immune-excluded area, and the score of the Immune-Desert area for the pathology slide image.

[0164] Accordingly, users can identify which genes are most highly expressed in iTILs and examine their genetic characteristics to determine which genetic traits are effective for cancer treatment. Furthermore, by identifying which genes possess genetic traits effective for cancer treatment, users can develop treatment methods for patients.

[0165] FIG. 9 is a diagram illustrating another example in which information about at least one biomarker according to one embodiment is output.

[0166] FIG. 9 illustrates a screen (900) on which information about biomarkers is displayed. For example, the processor (110) can display information corresponding to subtypes of multiple cells represented in a pathology slide image.

[0167] For example, the processor (110) can calculate the tumor subtype composition and immune subtype composition for each patient's sample. Accordingly, the processor (110) can identify an association in which a patient group with a high proportion of a specific subtype (e.g., Type 1) has a low response to therapy and a high rate of cancer recurrence. Based on this model, the processor (110) can predict the patient's response to therapy or prognosis based on the analysis of a new patient's sample and provide the results to the user.

[0168] Additionally, the processor (110) can generate a report of predicted medical information for each of the multiple patients or output it to the screen (900). For example, the processor (110) can generate a report of predicted medical information for each patient along with the identification information (e.g., ID) of each patient or output it to the screen (900).

[0169] If it is necessary to display predicted medical information separately from other patients (e.g., when treatment responsiveness is relatively high or low), the processor (110) can output the patient's identification information and / or medical information to the screen (900) separately from other patients (e.g., by different color, different icon, etc.).

[0170] FIG. 10 is a diagram illustrating an example in which information about at least one biomarker according to one embodiment is used to train a machine learning model.

[0171] The biomarker information described above with reference to FIGS. 1 to 9 can be used for training a machine learning model. That is, in order to improve the performance of the machine learning model, the processor (110) can generate new training data based on biomarker expression information.

[0172] For existing machine learning models, data directly labeled by experts (e.g., pathologists, doctors, etc.) on pathology slide images was used as training data. However, not only is the generation of such training data costly, but data quality issues may also arise due to variability among experts.

[0173] A processor (110) according to one embodiment can automatically generate a learning label (1020) based on the expression location of a biomarker. In this case, for example, the biomarker may include a marker gene that has been objectively verified through the spatial transcriptome analysis described above. For example, the learning label (1020) may be generated based on the expression information of a biomarker (e.g., at least one gene) at a specific location within a medical image and the information of an object corresponding to that location (e.g., type of cell, type of tissue, etc.). That is, the location where a specific gene or combination of genes is expressed may be considered as ground truth information representing the information of the object, and learning data (1030) may be constructed by matching this with a pathology slide image (1010).

[0174] Accordingly, a large amount of high-quality training data can be generated at a low cost. In addition, there is the advantage that machine learning models can learn even new cellular features that experts do not recognize.

[0175] In other words, as spatial transcriptome data at the single-cell level is utilized as training data, machine learning models can be trained by matching multiple genes for a single cell. Consequently, more detailed annotation results can be obtained compared to traditional expert annotation. Furthermore, machine learning models can be trained based on more accurate information combining a single cell and its corresponding gene expression data.

[0176] FIG. 11 is a drawing for illustrating an example of a system for outputting medical information according to one embodiment.

[0177] Referring to FIG. 11, the system (1100) is an example of a system and network for analyzing biomarkers using a machine learning model.

[0178] A scanner (1121), a user terminal (1122, 1123), an image management system (1130), an AI-based biomarker analysis system (1140), a laboratory information management system (1150), and / or a hospital or laboratory server (1160) can each be connected to a network (1170), such as the internet, via one or more computers, servers, and / or mobile devices, or can communicate with a user (1112) via one or more computers and / or mobile devices.

[0179] According to various embodiments of the present disclosure, the method described above with reference to FIGS. 2a through 10 may be performed by at least one of a user terminal (1122, 1123), an image management system (1130), an AI-based biomarker analysis system (1140), a laboratory information management system (1150), and a hospital or laboratory server (1160) or a combination thereof.

[0180] If the medical image is a pathology slide image, the scanner (1121) can obtain a digitized image from a tissue sample slide (pathology slide) created using a tissue sample of the patient (1111).

[0181] User terminals (1122, 1123), image management systems (1130), AI-based biomarker analysis systems (1140), laboratory information management systems (1150) and / or hospital or laboratory servers (1160) may generate or obtain from other devices tissue samples of one or more patients (1111), tissue sample slides (pathology slides), digitized images of tissue sample slides (pathology slides), various types of medical images of an object, or any combination thereof. Additionally, user terminals (1122, 1123), image management systems (1130), AI-based biomarker analysis systems (1140), laboratory information management systems (1150) and / or hospital or laboratory servers (1160) may obtain any combination of patient-specific information, such as the age, medical history, cancer treatment history, family history, past biopsy records, or disease information of one or more patients (1111).

[0182] A scanner (1121), a user terminal (1122, 1123), an AI-based biomarker analysis system (1140), a laboratory information management system (1150), and / or a hospital or laboratory server (1160) can transmit medical images, specific information of a patient (1111), and / or results of analyzing medical images to an image management system (1130) via a network (1170). The image management system (1130) may include a storage for storing received images and a storage device for storing analysis results.

[0183] In addition, according to various embodiments of the present disclosure, a machine learning model learned and trained to predict at least one of information regarding at least one cell, information regarding at least one region, and medical information (e.g., information related to biomarkers, medical diagnostic information, medical treatment information, etc.) from a medical image of a patient (1111) may be stored and operated in a user terminal (1122, 1123), an image management system (1130), etc.

[0184] As described above, pathological image information and spatial transcriptome information obtained from different image modalities are aligned based on the coordinates of cells detected by artificial intelligence, thereby enabling accurate integration of morphological information and gene expression information at the single-cell level.

[0185] In addition, by automatically identifying marker genes of specific cell types (e.g., tumor cells, immune cells, etc.) based on integrated data, new therapeutic targets can be discovered and diagnostic biomarkers developed.

[0186] In addition, genes associated with cancer malignancy can be identified based on the quantitative relationship between morphological characteristics, such as cell size and nucleus shape, and the expression levels of specific genes.

[0187] In addition, by analyzing differences in gene expression patterns (e.g., iTIL vs. sTIL) depending on the location of cells within the tissue structure (e.g., the tumor microenvironment), information on immune response and resistance mechanisms can be provided.

[0188] Furthermore, objective and scalable annotations based on gene expression information are generated and can be utilized for training machine learning models. Consequently, high-performance predictive models that go beyond the existing knowledge of pathologists can be developed.

[0189] In addition, personalized precision medicine can be realized by providing clinically important medical information through multifaceted analysis, such as predicting patient-specific treatment responsiveness or suggesting the likelihood of recurrence.

[0190] Meanwhile, the above-described method can be written as a program executable on a computer and can be implemented on a general-purpose digital computer that operates the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0191] A person skilled in the art related to the present embodiment will understand that it may be implemented in modified forms without departing from the essential characteristics of the description above. Therefore, the disclosed methods should be considered in an illustrative rather than a restrictive sense, and the scope of rights is defined in the claims rather than the description above, and should be interpreted to include all differences within the scope of equivalence.

Claims

1. At least one memory in which at least one instruction is stored; and It includes at least one processor that operates according to the above at least one instruction; and The above-mentioned at least one processor is, A computing device that identifies at least one object in a pathology slide image by analyzing the pathology slide image using a machine learning model, registers gene expression information corresponding to the at least one object included in spatial transcriptomics information with the at least one object expressed in the pathology slide image, and controls a display device to output information about at least one biomarker corresponding to the at least one object based on the result of the registration.

2. In Paragraph 1, The above-mentioned at least one processor is, A computing device that matches the gene expression information with the at least one object by matching the medical image in which the spatial transcriptome information is expressed with the pathology slide image.

3. In Paragraph 2, The above-mentioned at least one processor is, A computing device that matches the medical image and the pathology slide image based on at least one feature extracted from each of the pathology slide image and the medical image.

4. In Paragraph 2, The above-mentioned at least one processor is, A computing device that matches the medical image and the pathology slide image based on a first coordinate corresponding to at least one object expressed on the pathology slide image and a second coordinate corresponding to at least one object expressed on the medical image.

5. In Paragraph 1, The above-mentioned at least one processor is, A computing device that controls the display device to output at least one of the following: information about each of the plurality of cells expressed in the pathology slide image, information about at least one gene expressed in a cell specified by user input among the plurality of cells, or information about at least one gene expressed by each type of the plurality of cells.

6. In Paragraph 1, The above-mentioned at least one processor is, A computing device that controls the display device to output a plurality of cells represented in the above pathology slide image so that the plurality of cells are distinguished from each other by the subtypes in which the plurality of cells are classified.

7. In Paragraph 1, The above-mentioned at least one processor is, A computing device that controls a display device to output at least one of the following: information about each of a plurality of tissues expressed in the pathology slide image, information about at least one gene expressed in a tissue specified by user input among the plurality of tissues, or information about at least one gene expressed according to the type of the plurality of tissues.

8. In Paragraph 1, The above-mentioned at least one processor is, A computing device that controls the display device to output information corresponding to subtypes of multiple cells expressed in the pathology slide image.

9. In Paragraph 1, The above at least one object is, A computing device comprising at least one of the type of at least one cell expressed on the pathology slide image, the shape of the at least one cell, the shape of each of the components included in the at least one cell, or at least one type of at least one tissue expressed on the pathology slide image.

10. A step of identifying at least one object in a pathology slide image by analyzing the pathology slide image using a machine learning model; A step of registering gene expression information corresponding to at least one object included in spatial transcriptomics information with at least one object represented in the pathology slide image; and A method for outputting medical information, comprising the step of outputting information about at least one biomarker corresponding to at least one object based on the result of the above matching.

11. In Paragraph 10, The above matching step is, A method for matching the gene expression information and the at least one object by matching the medical image in which the spatial transcriptome information is expressed with the pathology slide image.

12. In Paragraph 11, The above matching step is, A method for matching a medical image and a pathology slide image based on at least one feature extracted from each of the pathology slide image and the medical image.

13. In Paragraph 11, The above matching step is, A method for matching a medical image and a pathology slide image based on a first coordinate corresponding to at least one object expressed on the pathology slide image and a second coordinate corresponding to at least one object expressed on the medical image.

14. In Paragraph 10, The above outputting step is, A method for outputting at least one of information regarding each of a plurality of cells expressed in the pathology slide image, information regarding at least one gene expressed in a cell specified by user input among the plurality of cells, or information regarding at least one gene expressed according to the type of the plurality of cells.

15. In Paragraph 10, The above outputting step is, A method for outputting a plurality of cells expressed in the above pathology slide image so that the plurality of cells are distinguished from one another according to the subtypes in which the plurality of cells are classified.

16. In Paragraph 10, The above outputting step is, A method for outputting at least one of information regarding each of a plurality of tissues expressed in the pathology slide image, information regarding at least one gene expressed in a tissue specified by user input among the plurality of tissues, or information regarding at least one gene expressed according to the type of the plurality of tissues.

17. In Paragraph 10, The above outputting step is, A method for outputting information corresponding to subtypes of multiple cells expressed in the above pathology slide image.

18. In Paragraph 10, The above at least one object is, A method comprising at least one of the type of at least one cell expressed on the pathology slide image, the shape of the at least one cell, the shape of each of the components included in the at least one cell, or at least one type of at least one tissue expressed on the pathology slide image.

19. A computer-readable recording medium storing a program for executing the method of claim 1 on a computer.

Images