System, method, and program for inferring disease-related condition

Abstract

This system for creating a machine learning model for inferring a disease-related condition comprises: a reception means for receiving training data, said training data including, for each of a plurality of subjects, an image of a hand of that subject, said image of the hand of each of the subjects being labeled with a disease-related condition of that subject, said image including an image, captured from the palm side, of the hand with at least one finger in a flexed state; and a creation means for creating a trained machine learning model through a training process using the training data.

Description

System, method, and program for estimating a state related to a disease

[0001] The present disclosure relates to a system, a method, and a program for estimating a state related to a disease. The present invention also relates to a system, a method, and a program for creating a machine learning model for estimating a state related to a disease.

[0002] For the diagnosis of diseases accompanied by external changes in body parts, such as acromegaly, efforts are being made to utilize machine learning of disease images and estimate diseases from the images (for example, Non-Patent Document 1).

[0003] ”Automatic Detection for Acromegaly Using Hand Photographs: A Deep-Learning Approach”, IEEE Access, Volume: 9, 23 December 2020

[0004] However, improvement in accuracy is required for the diagnosis of diseases using machine learning.

[0005] Furthermore, in order to improve the accuracy of diagnosis using machine learning, using a sufficient amount of learning data is one of the means. However, when using an image of a part of the human body as learning data for disease diagnosis, personal information (for example, fingerprints, face, etc.) may be included in the image. Therefore, it is difficult to increase such learning data, and it is difficult to secure a sufficient amount of learning data. This is because attention needs to be paid to the management of learning data so that the personal information of the image provider does not leak.

[0006] As a result of intensive research, the present inventors have found that the accuracy of estimating a state related to a disease can be improved by using an image taken from the palm side of a hand in a state where at least one finger is bent.

[0007] In addition, a learning data creation system has been developed that reduces the risk of data management by deleting areas containing personal information from hand images.

[0008] In one aspect, the system of the present disclosure comprises a receiving means for receiving training data, the training data comprising images of the hands of each of a plurality of subjects, the images of the subjects' hands being labeled with the subject's disease status, and the images of the hands including images of hands taken from the palm side with at least one finger bent, and a creation means for creating a trained machine learning model by training processing using the training data.

[0009] Furthermore, this disclosure provides, for example, the following items: (Item 1) A system for creating a machine learning model for estimating disease conditions, the system comprising: a receiving means for receiving training data, the training data comprising images of the hands of a plurality of subjects, the images of the subjects' hands being labeled with the disease conditions of the subjects, and the images comprising images of hands taken from the palm side with at least one finger bent; and a creation means for creating a trained machine learning model by training using the training data. (Item 2) The system further comprises preprocessing means for preprocessing training data, wherein the system includes detecting a first region and a second region in the image of each image of a plurality of subjects, the first region being a region corresponding to the personal information portion of the subject's hand, and the second region being a region corresponding to the important information portion of the subject's hand, determining whether the first region and the second region overlap, removing the image from the training data if the first region and the second region overlap, and masking the first region if the first region and the second region do not overlap, and the creation means performing a training process using the preprocessed training data, as described in the above item. (Item 3) The system according to any of the above items, wherein detecting the first region includes extracting the hand region in the image, extracting the hand skeleton in the extracted region, and detecting the first region based on the extracted skeleton. (Item 4) The system according to any of the above items, wherein the personal information portion includes fingerprints. (Item 5) A system according to any of the above items, wherein detecting a second region includes inputting an image into a second region detection machine learning model and detecting a second region based on the output from the second region detection machine learning model.(Item 6) The system according to any of the above items, wherein the second region detection machine learning model learns the relationship between pre-trained images of a hand and the state of the hand, and is configured to output an estimated state result from an input image, and detecting a second region based on the output from the second region detection machine learning model includes identifying a region in the image that at least influenced the estimation result, and detecting a second region based on the identified region. (Item 7) The system according to any of the above items, wherein the important information portion includes the fingertips. (Item 8) The system according to any of the above items, wherein the hand with at least one finger bent is a fist. (Item 9) The system according to any of the above items, wherein the image further includes an image taken from the back of the hand. (Item 10) The system according to any of the above items, wherein the disease is acromegaly. (Item 11) An estimation system for estimating the disease state of a subject, comprising: a receiving means for receiving subject data, wherein the subject data includes an image of the subject's hand, the image includes an image of the hand taken from the palm side with at least one finger bent; and an estimation means for estimating the subject's state based on the subject data. (Item 12) An estimation system according to any one of the above items, wherein the estimation means estimates the state using a machine learning model, and the machine learning model is a trained machine learning model created by the system described in any one of items 1 to 8.(Item 13) The estimation system further comprises processing means for preprocessing subject data, the preprocessing comprising: detecting a first region and a second region in an image, wherein the first region is a region corresponding to the personal information portion of the subject's hand, and the second region is a region corresponding to the important information portion of the subject's hand; determining whether the first region and the second region overlap; removing the image from the subject data if the first region and the second region overlap; and masking the first region if the first region and the second region do not overlap, the estimation means estimates the state based on the preprocessed subject data, the estimation system according to any of the above items. (Item 14) The estimation system according to any of the above items, wherein a hand with at least one finger bent is a fist. (Item 15) The estimation system according to any of the above items, wherein the image further comprises an image of a hand with all fingers extended, taken from the back of the hand. (Item 16) The disease is acromegaly, the estimation system described in any of the above items. (Item 17) A method for creating a machine learning model for estimating a disease condition, comprising: receiving training data, the training data comprising images of each subject's hand from a group of subjects, the images of the subjects' hands being labeled with the subject's disease condition, and the images comprising images of a hand taken from the palm side with at least one finger bent; and creating a trained machine learning model by training using the training data. (Item 18) A method for estimating a disease condition of a subject, comprising: receiving subject data, the subject data comprising images of the subject's hand, the images comprising images of a hand taken from the palm side with at least one finger bent; and estimating the subject's condition based on the subject data.(Item 19) A program for creating a machine learning model for estimating disease-related conditions, the program being executed on a computer having a processor, the program causing the processor to perform a process that includes receiving training data, the training data including images of each subject's hand from a plurality of subjects, the images of the subjects' hands being labeled with the subjects' disease-related conditions, and the images including images of a hand taken from the palm side with at least one finger bent, and creating a trained machine learning model by training using the training data. (Item 20) A program for estimating disease-related conditions of a subject, the program being executed on a computer having a processor, the program being caused the processor to perform a process that includes receiving subject data, the subject data including images of the subject's hand, the images including images of a hand taken from the palm side with at least one finger bent, and estimating the subject's condition based on the subject data. (Item 20A) A non-transient, computer-readable storage medium for storing the program described in Item 19 or Item 20. (Item 20B) A computer program product that includes a program as described in Item 19 or Item 20. (Item 21) A system for creating training data for machine learning, comprising: receiving means for receiving images of a plurality of subjects; preprocessing means for processing images, the preprocessing means being configured to perform the following operations for each image of a plurality of subjects: detecting a first region and a second region in the image, wherein the first region is a region corresponding to personal information and the second region is a region corresponding to important information; determining whether the first region and the second region overlap; removing the image from the training data if the first region and the second region overlap; and masking the first region if the first region and the second region do not overlap. (Item 22) The system as described in any of the above items, wherein the image is an image of a hand and the personal information portion is a fingerprint.

[0010] The system disclosed herein will enable the estimation of disease-related conditions with improved accuracy. Furthermore, the system disclosed herein can also provide machine learning models that enable such improved accuracy in condition estimation. This will lead to improvements in the fields of machine learning and medical imaging.

[0011] Furthermore, the systems described in this disclosure reduce the risks associated with managing training data that may contain personal information, and facilitate the preparation of a sufficient quantity of hand images for training data for various applications, such as estimating disease-related conditions. This also contributes to improvements in the field of machine learning.

[0012] Figure 1 shows an example of a flow chart 10 for estimating a disease-related condition using one embodiment of the present disclosure.

[0013] Figure 2 shows an example of the configuration of a system 100 for estimating disease-related conditions.

[0014] Figure 3A shows an example of the configuration of the processor unit 120 of the system 100 in one embodiment.

[0015] Figure 3B shows an example of the configuration of the processor unit 120' of system 100 in another embodiment.

[0016] Figure 3C shows an example of the configuration of the processor unit 140 of the system 100 in one embodiment.

[0017] Figure 3D shows an example of the configuration of the processor unit 140' of system 100 in another embodiment.

[0018] Figure 4A shows a flowchart 400 for removing personal information from an image of a subject's hand in one embodiment.

[0019] Figure 4B shows a flowchart 410 for removing personal information from an image of a subject's hand in one embodiment.

[0020] Figure 5A is a flowchart showing an example of the process (process 500) of creating a machine learning model for estimating disease-related conditions using system 100.

[0021] Figure 5B is a flowchart showing another example (process 510) of the process by which system 100 creates a machine learning model to estimate disease-related conditions.

[0022] Figure 6A is a flowchart showing an example of a process (process 600) by system 100 for estimating a disease-related state.

[0023] Figure 6B is a flowchart showing another example (process 610) of the process by which system 100 estimates the state of a disease.

[0024] The present disclosure is described below. Throughout this specification, singular expressions should be understood to include the concept of their plural form unless otherwise specified. Thus, singular articles (for example, "a," "an," and "the" in English) should be understood to include the concept of their plural form unless otherwise specified. Furthermore, terms used herein should be understood to have the meaning commonly used in the art unless otherwise specified. Thus, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. In case of any conflict, this specification (including definitions) shall prevail.

[0025] (Definitions) In this specification, “Subject” means any entity that provides training data for the technology of this disclosure to create a machine learning model.

[0026] In this specification, “Subject” means any entity that the technology of this disclosure uses to estimate a disease-related condition.

[0027] In this specification, "approximately" means ±10% of the following number.

[0028] In this specification, "approximately perpendicular" means a state in which the angle between two lines or planes is 90° ± 10°.

[0029] 1. Flowchart 1 for estimating disease-related conditions shows an example of a flow 10 for estimating a subject's disease-related condition from an image of the subject's hand, using one embodiment of the present disclosure. Flowchart 10 also describes a learning stage prior to estimating the subject's disease-related condition. This learning stage provides a trained machine learning model capable of estimating the subject's disease-related condition, and in flowchart 10, the estimation is performed using the trained machine learning model provided in the learning stage. Following flowchart 10, subject T can learn about their disease-related condition simply by taking an image of their hand with the terminal device 300.

[0030] In the learning phase, step S0 provides learning data to system 100. System 100 is, for example, a server device. The manner in which the learning data is provided to system 100 is not limited. For example, learning data may be provided to system 100 via a network (e.g., the Internet, LAN, etc.). For example, learning data may be provided to system 100 via a storage medium (e.g., removable media).

[0031] The training data includes images of the hands of multiple subjects, and each subject's hand image is labeled with the subject's disease status (e.g., normal, or mild, moderate, or severe disease status). Here, the disease is typically a disease that presents with a cosmetic abnormality of the hands or nails, and includes, but is not limited to, acromegaly, Cushing's syndrome, pseudohypoparathyroidism, rheumatoid arthritis, lung cancer, anemia, osteoarthritis, scleroderma, and psoriatic arthritis. In a preferred embodiment, the disease in the present invention may be acromegaly.

[0032] An image of a hand may be an image obtained by photographing a hand in a predetermined state from a predetermined direction. The predetermined state of the hand and the predetermined direction for photography may be any as appropriate depending on the diagnosis of the disease. Preferably, the predetermined state of the hand is one in which the visual difference of the hand or nails is clearly evident depending on the presence or degree of the disease. Specifically, the predetermined state of the hand may be one in which at least one finger is bent, all fingers are bent, a fist is made, or all fingers are extended. Preferably, the predetermined direction for photography is one in which the visual difference of the hand or nails is clearly visible in the image. Specifically, the predetermined direction for photography may be approximately perpendicular to the palm or approximately perpendicular to the back of the hand.

[0033] In one example, the image of the hand includes an image of a hand with at least one finger bent, taken from the palm side. Preferably, the bent finger may be a clenched fist.

[0034] In one example, the hand image includes an image of a hand with all fingers extended, taken from the back of the hand.

[0035] In one example, the hand images include images of a hand with at least one finger bent, taken from the palm side, as well as images of a hand taken from the back side, or images of a hand with all fingers extended, taken from the back side. The hand images may be further labeled by a particular hand position and / or orientation.

[0036] The predetermined hand position and orientation for photography can be determined based on the principle that the subject's personal information (e.g., fingerprints) does not appear in the image. For example, a clenched fist is preferred because it is a hand position in which at least the fingerprints of the second to fifth fingers do not appear in the image. For example, a hand with all fingers extended is preferred because, when photographed from the back of the hand, the fingerprints of all fingers do not appear in the image.

[0037] For example, if a subject's personal information is present in the image, the system 100 may preprocess the input training data to remove the personal information. The preprocessed training data is then used for training the pre-trained machine learning model. This allows the pre-trained machine learning model to learn the relationship between hand images and the labels attached to them, and as a result, the trained machine learning model will, at the very least, output a disease-related condition corresponding to an image when an image is input. The trained machine learning model is then used in subsequent flows.

[0038] Subject T takes an image of their hand with the terminal device 300 in order to estimate the condition of their disease. The hand image is obtained by taking an image of the hand in a predetermined state from a predetermined direction, similar to the subject image described above. For example, if when subject T takes an image of their hand, the hand is not in a predetermined state, and / or the shooting direction is outside the predetermined direction or the hand is not within the field of view of the terminal device 300, the terminal device 300 may provide subject T with a visual and / or auditory warning. Alternatively, the terminal device 300 may provide subject T with a template or instructions to guide subject T to ensure that their hand is in a predetermined state or that the shooting direction is aligned with a predetermined direction, in order to facilitate the shooting process.

[0039] Once an image of subject T is acquired, in step S1, subject data, including an image of the hand, is input to the system 100. Subject data can be input to the system 100 in any manner. For example, it may be input to the system 100 from a terminal device via a network (e.g., the Internet, LAN, etc.), or it may be stored in a storage medium in the terminal device and then input to the system 100 when the storage medium is connected to the system 100.

[0040] For example, when the personal information of the subject T appears in an image, the system 100 can perform preprocessing to remove the personal information included in the input subject data. Thereafter, the system 100 inputs the subject data into the learned machine learning model to estimate the state related to the disease.

[0041] In step S2, the result estimated by the system 100 is provided to the subject T. The mode in which the estimated result is provided is not limited. For example, the estimated result may be provided from the system 100 to the terminal device 300 via a network, may be provided to the subject T via a storage medium, or may be provided to the subject T via a paper medium.

[0042] Thereby, the subject T can simply know whether he / she has a disease or the level of his / her disease.

[0043] In addition to or instead of step S2, in step S3, the result estimated by the system 100 is provided to a doctor. The mode in which the estimated result is provided is not limited. For example, the estimated result may be provided from the system 100 to the terminal device of the hospital H via a network, may be provided to the doctor via a storage medium, or may be provided to the doctor via a paper medium. The estimated result can be provided as digital data.

[0044] Thereby, the doctor can utilize the estimated result for diagnosing whether the subject T has a disease or the level of the disease that the subject T has. For example, even for a disease that is difficult to diagnose or requires experience or knowledge for diagnosis, it may be possible to accurately diagnose by using the estimated result.

[0045] System 100 can automatically estimate the state related to a disease and provide the estimated result in response to the input of an image of subject T. Alternatively, System 100 can automatically estimate the state related to a disease and provide the estimated result in response to the acquisition of an image of subject T. That is, the above-described flow 10 by System 100 can be automatically performed by System 100.

[0046] In the learning process of the machine learning model used in the above-described flow 10, personal information that may be included in the learning data is removed, and personal information of subject T can also be removed in the process of estimating the state of subject T. This enables the protection of the privacy and the reduction of the psychological burden of the subjects and the subjects, and can accelerate the social implementation of a diagnostic system utilizing machine learning. Furthermore, in the above-described flow 10, it is possible to achieve high accuracy even in the estimation of diseases with a low morbidity rate and high specialization such as acromegaly. Early detection of such diseases can be expected to efficiently lead to a visit to an appropriate specialist.

[0047] In the above-described example, the flow 10 in which subject T can know the state related to his / her own disease only by taking a picture of his / her hand with the terminal device 300 has been described, but the present invention is not limited thereto. For example, even if a doctor takes a picture of subject T's hand using a camera, or subject T asks another person to take a picture of his / her hand using the terminal device 300, a flow in which the estimation result of the state of subject T's disease is provided to a doctor, a physical therapist, a caregiver, or the family of subject T, etc. is also possible.

[0048] In the above-described example, it has been described that System 100 includes both the learning system 11 and the estimation system 12, but System 100 may not include the learning system 11. For example, the estimation system 12 of System 100 may use a learned machine learning model provided from outside System 100 to estimate the state of subject T's disease.

[0049] Furthermore, the processing performed by the system 100 described above can also be carried out by the terminal device 300. In this case, the system 100 and the terminal device 300 can be integrated. In the example described above, where the estimated results of the disease status of subject T are provided to a doctor, physical therapist, caregiver, or family member of subject T, the estimated results can be provided directly from the terminal device 300.

[0050] Although Figure 1 shows an image of a hand in a single state, multiple types of hand images may be used. Multiple types of hand images may include, for example, an image of a hand with at least one finger bent, taken from the palm side, and an image of a hand taken from the back side, and more preferably, an image of a clenched fist, taken from the palm side, and an image of an open hand, taken from the back side.

[0051] The system 100 described above may have the configuration described later.

[0052] 2. Diagram 2 of the system configuration for estimating disease-related conditions shows an example of the configuration of the system 100 for estimating disease-related conditions.

[0053] In this example, system 100 is connected to the database unit 200. Furthermore, system 100 is connected to at least one terminal device 300 via a network N.

[0054] Network N can be any type of network. Network N may be, for example, the Internet or a LAN. Network N may be a wired network or a wireless network.

[0055] An example of system 100 is a server device, but is not limited thereto. An example of terminal device 300 is a terminal device held by a subject or test subject, a terminal device installed in a hospital, or a terminal device installed in a public place (e.g., a community center, government office, library, etc.), but is not limited thereto. Here, the server device and terminal device can be any type of computer. For example, the terminal device can be any type of terminal device such as a smartphone, tablet, personal computer, or smart glasses. Preferably, terminal device 300 is equipped with or connected to a camera or other means of capturing images. The camera or other means of capturing images may be, for example, a dedicated camera configured to acquire only images of the hand parts necessary for processing by system 100. The dedicated camera may be controlled to acquire images only when the hand parts to be acquired are in the field of view. System 100 comprises an interface unit 110, a processor unit 120, and a memory unit 130.

[0056] The interface unit 110 exchanges information with the outside of the system 100. The processor unit 120 of the system 100 can receive information from the outside of the system 100 via the interface unit 110 and can transmit information to the outside of the system 100. The interface unit 110 can exchange information in any format.

[0057] The interface unit 110 includes, for example, an input unit that enables information to be input to the system 100. The manner in which the input unit enables information to be input to the system 100 is not limited. For example, if the input unit is a receiver, information may be input by the receiver receiving information from outside the system 100 via the network N. Alternatively, if the input unit is a data reading device, information may be input by reading information from a storage medium connected to the system 100. Alternatively, if the input unit is a touch panel, information may be input by the user touching the touch panel. Alternatively, if the input unit is a mouse, information may be input by the user operating the mouse. Alternatively, if the input unit is a keyboard, information may be input by the user pressing keys on the keyboard. Alternatively, if the input unit is a microphone, information may be input by the user inputting voice into the microphone. Alternatively, if the input unit is a camera, information captured by the camera may be input.

[0058] For example, the input unit enables input of learning data into system 100. For example, the input unit enables input of subject data into system 100.

[0059] The interface unit 110 includes, for example, an output unit that enables information to be output from the system 100. The mode by which the output unit enables information to be output from the system 100 is not limited. For example, if the output unit is a transmitter, the information may be output by the transmitter transmitting information to the outside of the system 100 via the network N. Alternatively, if the output unit is a data writing device, the information may be output by writing the information to a storage medium connected to the system 100. Alternatively, if the output unit is a display screen, the information may be output to the display screen. Alternatively, if the output unit is a speaker, the information may be output by sound from the speaker.

[0060] For example, the output unit enables the system 100 to output the subject's state estimated by the system 100 to an external source. For example, the output unit enables the system 100 to output the machine learning model created by the system 100 to an external source. For example, the output unit enables the system 100 to output the training data created by the system 100 (e.g., training data from which personal information has been removed) to an external source.

[0061] The processor unit 120 executes the processing of the system 100 and controls the operation of the entire system 100. The processor unit 120 reads the program stored in the memory unit 130 and executes the program. This makes it possible to make the system 100 function as a system that executes desired steps. The processor unit 120 may be implemented by a single processor or by multiple processors.

[0062] The memory unit 130 stores programs necessary for executing the processes of the system 100, as well as data necessary for executing those programs. The memory unit 130 may store programs for processes for creating machine learning models that estimate disease-related conditions (for example, programs that implement the processes shown in Figure 5A or Figure 5B described later) and / or programs for processes for estimating disease-related conditions (for example, programs that implement the processes shown in Figure 6A or Figure 6B described later). The memory unit 130 may also store programs for processes for creating training data for machine learning. Here, it is not relevant how the programs are stored in the memory unit 130. For example, the programs may be pre-installed in the memory unit 130. Alternatively, the programs may be installed in the memory unit 130 by being downloaded via a network. Alternatively, the programs may be stored on a computer-readable storage medium. Alternatively, the programs may be provided as a program product that includes the programs.

[0063] The database unit 200 may store, for example, training data including images of the hands of multiple subjects. The training data may be transmitted from the terminal device 300 to the database unit 200 (via the system 100), or it may be captured by a camera that the system 100 may have. The hand image of each subject of the multiple subjects may be stored in association with the disease status of each subject. Furthermore, the hand image of each subject of the multiple subjects may be stored in association with a specific hand state and / or a specific shooting direction. The data stored in the database unit 200 may be used, for example, to create a machine learning model for estimating diseases. The training data stored in the database unit 200 may be data that has been processed to remove personal information.

[0064] Furthermore, the database unit 200 may store, for example, the estimated results of disease-related conditions output by the system 100.

[0065] In the example shown in Figure 2, the database unit 200 is located outside the system 100, but the present invention is not limited to this. It is also possible to provide at least a portion of the database unit 200 inside the system 100. In this case, at least a portion of the database unit 200 may be implemented by the same storage means as the storage means implementing the memory unit 130, or by storage means different from the storage means implementing the memory unit 130. In any case, at least a portion of the database unit 200 is configured as a storage unit for the system 100. The configuration of the database unit 200 is not limited to a specific hardware configuration. For example, the database unit 200 may be composed of a single hardware component or of multiple hardware components. For example, the database unit 200 may be configured as an external hard disk drive for the system 100, or as storage on the cloud connected via a network N.

[0066] Figure 3A shows an example of the configuration of the processor unit 120 in one embodiment. The processor unit 120 may have a configuration for processing to create a machine learning model for estimating disease-related conditions.

[0067] The processor unit 120 includes a receiving means 121 and a creation means 122.

[0068] The receiving means 121 is configured to receive training data. The receiving means 121 can receive training data received from outside the system 100, for example, via the interface unit 110. The receiving means 121 may, for example, receive training data from a terminal device 300 via the interface unit 110, receive training data from a database unit 200 via the interface unit 110, or receive training data from other sources via the interface unit 110.

[0069] The training data includes images of the hands of multiple subjects, and each subject's hand image is labeled with the subject's disease status (e.g., normal, or mild, moderate, or severe disease status). For example, the disease status label may be "0" indicating no disease or normal status, "1" indicating disease present or abnormal status, or a numerical value between 0 and 1 depending on the severity of the disease. The hand images are acquired by photographing a hand in a predetermined state from a predetermined direction. The hand images may be labeled by a predetermined hand state and predetermined direction. The hand images may all be images acquired by photographing a single hand in a single predetermined state from a single predetermined direction, or they may include images acquired by photographing hands in different predetermined states from different predetermined directions. The predetermined state and predetermined direction may be any one that corresponds to the diagnosis of the disease. Preferably, the predetermined state is a hand condition in which a noticeable difference in the appearance of the hand or nails is apparent depending on the presence or degree of disease. Specifically, the predetermined state of the hand may be a hand with at least one finger bent, a hand with all fingers bent, a hand in a fist, or a hand with all fingers extended. Preferably, the predetermined direction for photography is a direction in which a noticeable difference in the appearance of the hand or nails is clearly visible in the image. Specifically, the predetermined direction may be approximately perpendicular to the palm or approximately perpendicular to the back of the hand. In one example, the hand image includes an image of a hand with at least one finger bent, taken from the palm side, preferably a hand with at least one finger bent, which may be a fist. In one example, the hand image includes an image of a hand with all fingers extended, taken from the back of the hand. In one example, the hand image includes an image of a hand with at least one finger bent, taken from the palm side, as well as an image of a hand taken from the back of the hand or an image of a hand with all fingers extended, taken from the back of the hand. The predetermined hand position and shooting direction can be determined based on the principle that the subject's personal information (e.g., fingerprints) does not appear in the image. For example, a clenched fist is a hand position in which at least the fingerprints of the second to fifth fingers do not appear in the image.For example, a hand with all fingers extended is a hand condition in which fingerprints of all fingers do not appear in the image when photographed from the back of the hand. In one example, to create a machine learning model capable of estimating the condition of acromegaly, images of the hands of multiple subjects may include images of the hands of subjects with acromegaly and images of the hands of subjects without acromegaly. Images of the hands of subjects with acromegaly may be labeled, for example, as positive or as indicating the degree of the acromegaly disease. Images of the hands of subjects without acromegaly may be labeled, for example, as negative.

[0070] In one example, images of the hands of multiple subjects may include, for each of several diseases, images of the hands of subjects with the disease and images of the hands of subjects without the disease. Such images can be used to create a machine learning model capable of estimating the state of multiple diseases. For example, to create a machine learning model capable of estimating the state of two diseases, images of the hands of multiple subjects may include images of the hands of subjects with a first disease, images of the hands of subjects without the first disease, images of the hands of subjects with a second disease, and images of the hands of subjects without the second disease. The images of the hands of subjects with the first disease may be labeled, for example, with a label indicating positivity for the first disease or a label indicating the severity of the first disease. The images of the hands of subjects without the second disease may be labeled, for example, with a label indicating negativity for the second disease. The images of the hands of subjects with the second disease may be labeled, for example, with a label indicating positivity for the second disease or a label indicating the severity of the second disease. The images of the hands of subjects without the second disease may be labeled, for example, with a label indicating negativity for the second disease.

[0071] The learning data received by the receiving means 121 is provided to the creation means 122.

[0072] The creation means 122 is configured to create a trained machine learning model by training a pre-training machine learning model with the relationship between each subject's hand image and each subject's disease-related condition. Training a pre-training machine learning model with the relationship between each subject's hand image and each subject's disease-related condition can be done using methods that are publicly known or may become publicly known in the art.

[0073] For example, a machine learning model could be a neural network model, and more specifically, a convolutional neural network (CNN). In one example, the machine learning model could be a ResNet-based model, and more specifically, a ResNet-50-based model.

[0074] For example, when training a neural network model with training data, an image of a hand may be applied to the input layer, and a label indicating the disease state attached to that image may be applied to the output layer. That is, a pair of input data for training and output data may be (image of a hand, label indicating the disease state attached to the hand image), for example, (image of the hand of the first subject, label indicating the disease state of the first subject), (image of the hand of the second subject, label indicating the disease state of the second subject), ... (image of the hand of the nth subject, label indicating the disease state of the nth subject).

[0075] The creation means 122 may also create a trained machine learning model by fine-tuning an already created machine learning model. This is preferable because it allows for the creation of a trained machine learning model specialized for a specific group of subjects.

[0076] The trained machine learning model created in this way has learned the relationship between images of each subject's hands and their disease status for multiple subjects. Therefore, when an image of a subject's hand is input, it can output the subject's disease status. The output could be, for example, a score indicating the probability of having a particular disease. Alternatively, the output could be, for example, a score indicating the level of a particular disease.

[0077] The creation means 122 may be configured to create a trained machine learning model by also utilizing predetermined states of the hand and predetermined orientations for photography, which can be labeled to the hand images in the training data. That is, the creation means 122 can create a trained machine learning model by training a pre-trained machine learning model on the relationship between each subject's hand image, predetermined hand states, predetermined orientations for photography, and the disease state of each subject. For example, when training a neural network model with training data, the input layer may be applied to the hand image and labels indicating predetermined states and orientations, and the output layer may be applied to the label indicating the disease state attached to the image. In other words, a pair of input data for learning and output data may be ([image of a hand, a label indicating a predetermined state and direction attached to the image of the hand], a label indicating the state of the disease attached to the image of the hand), for example, ([image of the hand of the first subject, a label indicating a predetermined state and direction attached to the image of the first subject], a label indicating the state of the disease of the first subject), ([image of the hand of the second subject, a label indicating a predetermined state and direction attached to the image of the second subject], a label indicating the state of the disease of the second subject), ... ([image of the hand of the nth subject, a label indicating a predetermined state and direction attached to the image of the nth subject], a label indicating the state of the disease of the nth subject). The trained machine learning model created in this way has learned the relationship between each subject's hand image, a predetermined state of the hand, a predetermined direction for photography, and the state of each subject's disease for multiple subjects, so when a subject's hand image, a predetermined state of the hand, and a predetermined direction for photography are input, the model can output the state of the subject's disease.

[0078] The creation means 122 may be configured to create a trained machine learning model by training a pre-trained machine learning model on the relationship between each subject's hand image and each subject's status related to multiple diseases. The trained machine learning model thus created has learned the relationship between each subject's hand image and each subject's status related to multiple diseases for multiple subjects, so when a subject's hand image is input, it can output each of the subject's status related to multiple diseases. The output may be, for example, a score indicating the probability of each of the multiple diseases being present. Alternatively, the output may be, for example, a score indicating the level of each of the multiple diseases.

[0079] The creation means 122 may be configured to create a trained machine learning model by training a pre-training machine learning model on the relationship between multiple types of hand images of each subject and the disease state of each subject. The multiple types of hand images include, for example, an image of a hand with at least one finger bent, taken from the palm side, and an image of a hand taken from the back side, and more preferably, an image of a clenched fist, taken from the palm side, and an image of an open hand, taken from the back side.

[0080] Figure 3B shows an example of the configuration of a processor unit 120', which is an alternative embodiment of the processor unit 120. The processor unit 120' may have the same configuration as the processor unit 120, except that it includes a preprocessing means 123. The system 100 may have a processor unit 120' instead of the processor unit 120. In Figure 3B, components having the same configuration as those described above with reference to Figure 3A are given the same reference numerals, and detailed explanations are omitted here.

[0081] The processor unit 120' includes a receiving means 121, a preprocessing means 123, and a creation means 122.

[0082] The receiving means 121 is configured to receive training data. The training data received by the receiving means 121 is provided to the preprocessing means 123.

[0083] The preprocessing means 123 is configured to perform preprocessing to remove personal information from hand images included in the input training data.

[0084] Figure 4A schematically illustrates the flow 400 in which personal information is removed from a hand image by the preprocessing means 123.

[0085] The preprocessing means 123 is provided with images of multiple subjects' hands from the receiving means 121. The preprocessing means 123 performs the following steps for each subject's hand image.

[0086] In step S401, the preprocessing means 123 detects a first region and a second region in the image of each subject's hand. The first region is a region corresponding to a personal information portion containing the subject's personal information. The personal information portion is, for example, a fingerprint, but is not limited to that. The second region is a region corresponding to an important information portion that is important for estimating the state of a disease. The important information portion may be, for example, a predetermined part where the presence or degree of a disease makes a significant difference in appearance. The important information portion may be, for example, the fingertips, nails, or other fingertips when creating a machine learning model capable of estimating the state of acromegaly, but is not limited to that. As an example, the important information portion may be a part of the hand corresponding to a region that has a significant impact on the estimation result of the second region detection machine learning model according to the present invention, and such a part of the hand may be, for example, the joints of the fingers and / or the area around the thenar eminence. The detection of the first and second regions by the preprocessing means 123 can be carried out using methods that are publicly known or may become publicly known in the art.

[0087] An example of the processing performed by the preprocessing means 123 to detect a first region is described below. In one example, the preprocessing means 123 can detect a first region by performing a region extraction process to extract the hand region from the image and a skeletal extraction process to extract the hand skeleton. The hand region extraction process is, for example, a process that classifies each pixel in the hand image into either the hand region or another region. The hand region extraction process includes, but is not limited to, a segmentation process. The hand region extraction process may be implemented using a known CNN that has been trained to extract and output the hand region when an image of a hand is input.

[0088] Skeleton extraction is a process that converts, for example, the hand region into a skeleton, which is a combination of lines one pixel wide. Skeleton extraction can be implemented, for example, by using the skeletonization method (https: / / scikit-image.org / docs / 0.24.x / auto_examples / edges / plot_skeleton.html), but is not limited to this.

[0089] The preprocessing means 123 can detect a first region based on the extracted skeleton. For example, the preprocessing means 123 can identify the fingertip position (i.e., the tip of the extracted skeleton) based on the extracted skeleton and detect a first region based on this. For example, the preprocessing means 123 can identify the fingertip position of an unbent finger or the thumbtip position and detect a first region (e.g., the fingertip area) based on the identified fingertip position. For example, a predetermined range of the area from the fingertip position can be detected as the first region. In addition to the identified tip position, the preprocessing means 123 may also detect a first region based on the hand region extracted by the method described above. As an example, the preprocessing means 123 can detect a common area between the predetermined range of the fingertip position and the hand region extracted in the region extraction process as the first region.

[0090] By performing hand region extraction processing and skeletal extraction processing, the preprocessing means 123 can accurately detect the first region.

[0091] An example of the processing performed by the preprocessing means 123 to detect the second region is described below. For example, the preprocessing means 123 can detect the second region by using a machine learning model trained to detect the second region (hereinafter referred to as the "second region detection machine learning model").

[0092] As an example, a second region detection machine learning model could be a machine learning model that has learned regions corresponding to important information parts in an image. Such a second region detection machine learning model can detect and output regions corresponding to predetermined important information parts (e.g., fingertips, nails, etc.) when an image is input. This is useful, for example, when the important information parts are known.

[0093] As another example, the second region detection machine learning model could be a machine learning model that has been pre-trained on the relationship between pre-trained images of hands and the state of those hands, similar to the machine learning model created by the creation means 122 described above. The pre-trained images may be part of the training data received by the receiving means 121, or they may be data separate from the training data received by the receiving means 121. For example, when the pre-trained images are at least part of the training data, the input data and output data pairs for pre-training may be (pre-trained hand image, label indicating the state of the disease attached to the pre-trained hand image), for example, (pre-trained hand image of the first subject, label indicating the state of the disease of the first subject), (pre-trained hand image of the second subject, label indicating the state of the disease of the second subject), ... (pre-trained hand image of the Nth subject, label indicating the state of the disease of the Nth subject) (where N ≤ n). From the standpoint of protecting personal information, pre-training is preferably performed using pre-training data that includes images of hands with the first region masked. Masking is performed, for example, by filling in at least a portion of the first region with a specific color. Such a second region detection machine learning model can output an estimated result of the state corresponding to the image when an image is input. The pre-processing means 123 can detect the region that influenced the estimation result by the second region detection machine learning model as the second region. Identifying regions that influenced the estimation results can be performed by a process that includes, for example, at least one of CAM (Class Activation Mapping), Grad-CAM (Gradient-weighted Class Activation Mapping), score-CAM (score-weighted Class Activation Mapping), or Group score-weighted Class Activation Mapping. As an example, identifying regions that influenced the estimation results may involve calculating the magnitude of the influence on the estimation results for each small region in the image (e.g., each pixel) and identifying regions where the magnitude of the influence exceeds a predetermined threshold.Regions identified as having at least influenced the estimation results can be visualized, for example, using a heatmap that represents the magnitude of the influence using the intensity of a specific color.

[0094] The second region detection machine learning model can preferably be implemented in the terminal device 300. This allows the processing for detecting the second region to be contained within the terminal device 300, eliminating the need for images containing the subject's personal information to be transmitted outside the terminal device 300, and further reducing data management risks.

[0095] For example, the detection of the second region may be performed using known machine learning models and / or known skeleton extraction techniques that detect specific regions by focusing on various features (such as color information or brightness information).

[0096] The preprocessing means 123 may determine the first and / or second regions based on user input, instead of detecting the first and / or second regions, or in addition to detecting the first and / or second regions. The manner in which the user input is provided to the preprocessing means 123 is irrelevant. The user input may be provided from outside the system 100 via the interface unit 110 of the system 100, for example, from a terminal device 300. The user input may include, for example, selecting the first and / or second regions in an image of a hand or region of a hand displayed on the terminal device 300. The user input may include, for example, modifying the first and / or second regions displayed on the terminal device 300.

[0097] In step S402, the preprocessing means 123 determines whether the first region and the second region overlap. Whether or not they overlap may be determined by whether or not at least a part of the first region and the second region overlap, or by using a threshold value that is set as appropriate. For example, whether or not they overlap may be determined by whether or not the ratio of the area of ​​the overlapping region to the sum of the areas of the first region and the second region exceeds a certain threshold value.

[0098] If the first region and the second region do not overlap (for example, in the upper center image of Figure 4A), in step S403, the preprocessing means 123 masks the first region. Masking is performed, for example, by filling in at least a portion of the first region with a specific color. In the example shown in Figure 4A, the portion of the first region corresponding to the fingerprint is filled in black.

[0099] If the first region and the second region overlap (for example, the image in the lower center of Figure 4A), in step S404, the preprocessing means 123 removes that image from the training data. If the personal information portion is masked as in step S403, the important information portion is also masked, rendering the image worthless as training data. Even if an image with such important information portion masked is used for training, high-accuracy estimation cannot be expected.

[0100] As shown in Figure 4A, the preprocessing means 123 can remove personal information from the hand image through the process described above.

[0101] Although Figures 3B and 4A illustrate the processing steps S401 to S404 performed by the preprocessing means 123, the present invention is not limited thereto. The processing steps S401 to S404 may, for example, be performed by the terminal device 300 before the subject data is transmitted to the system 100. This allows the system 100 to create a machine learning model that estimates disease-related conditions without receiving images containing personal information, further reducing data management risks.

[0102] Referring again to Figure 3B, the training data preprocessed by the preprocessing means is provided to the creation means 122.

[0103] The creation means 122 is configured to create a trained machine learning model by training a pre-trained machine learning model on the relationship between pre-processed images of each subject's hand and the disease state of each subject.

[0104] The trained model created in this way can be used, for example, in the processor unit 140 described later. Furthermore, the parameters of the trained model created in this way can be stored in the database unit 200.

[0105] As described above, system 100 may have a processor unit 120 or processor unit 120' for processing to create a machine learning model for estimating disease-related conditions. In addition to or instead of processor unit 120 or processor unit 120', system 100 may have a processor unit 140 for processing to estimate disease-related conditions.

[0106] Figure 3C shows an example of the configuration of the processor unit 140. The processor unit 140 may have a configuration for processing to estimate the state related to the disease.

[0107] The processor unit 140 includes a receiving means 141 and an estimation means 142.

[0108] The receiving means 141 is configured to receive subject data. The receiving means 141 can receive subject data received from outside the system 100, for example, via the interface unit 110. The receiving means 121 may, for example, receive subject data from a terminal device 300 via the interface unit 110, receive subject data from a database unit 200 via the interface unit 110, or receive subject data from other sources via the interface unit 110.

[0109] The subject data includes images of the subject's hands. The hand images are obtained by photographing a hand in a predetermined state from a predetermined direction. The predetermined state and direction can be any as appropriate depending on the diagnosis of the disease. Preferably, the predetermined state is a hand condition in which a significant difference in the appearance of the hand or nails is evident depending on the presence or degree of the disease. Specifically, the predetermined state of the hand may be a hand with at least one finger bent, all fingers bent, a hand in a fist, or all fingers extended. Preferably, the predetermined direction for photography is a direction in which a difference in the appearance of the hand or nails is clearly visible in the image. Specifically, the predetermined direction may be approximately perpendicular to the palm or approximately perpendicular to the back of the hand. In one example, the hand image includes an image of a hand with at least one finger bent, photographed from the palm side. Preferably, the hand with at least one finger bent may be a hand in a fist. In another example, the hand image includes an image of a hand with all fingers extended, photographed from the back of the hand. In one example, the hand images include an image of a hand with at least one finger bent, taken from the palm side, as well as an image of a hand taken from the back side, or an image of a hand with all fingers extended, taken from the back side. The hand images may be further labeled by a predetermined hand state and / or predetermined shooting direction. The predetermined hand state and predetermined shooting direction may be determined on the basis that the subject's personal information (e.g., fingerprints) does not appear in the image. For example, a clenched fist is a hand state in which at least the fingerprints of the second to fifth fingers do not appear in the image. For example, all fingers extended is a hand state in which, when photographed from the back side, the fingerprints of all fingers do not appear in the image.

[0110] Preferably, the predetermined state and direction in the subject data are the same as the predetermined state and direction in the training data. This enables more accurate estimation by the estimation means 142 described later.

[0111] The target data received by the receiving means 141 is provided to the estimation means 142.

[0112] The estimation means 142 is configured to estimate the subject's disease condition based on an image of the subject's hand.

[0113] The estimation means 142 can estimate, for example, whether the subject has a certain disease or not, as a state relating to a certain disease. Alternatively, in addition to or instead of the above, the estimation means 142 can estimate the level of a certain disease the subject has (e.g., mild, moderate, severe), as a state relating to a certain disease. Alternatively, in addition to or instead of the above, the estimation means 142 can estimate which of several diseases the subject has, as a state relating to multiple diseases. Alternatively, the estimation means 142 can estimate the state relating to a disease based on multiple images of the subject's hands. Alternatively, the estimation means 142 can estimate the state relating to a disease based on multiple types of images of the subject's hands. Multiple types of hand images include, for example, an image of a hand with at least one finger bent, taken from the palm side, and an image of a hand taken from the back side, and more preferably, an image of a clenched fist taken from the palm side, and an image of an open hand taken from the back side.

[0114] The results estimated by the estimation means 142 can be output to the outside of the system 100 via the interface unit 110. For example, the estimated results can be transmitted to the subject's terminal device 300 via the interface unit 110. This allows the subject to check their own status via their terminal device 300. For example, the estimated results can be transmitted to the doctor's terminal device 300 via the interface unit 110. This allows the doctor to use the estimated results to help diagnose the subject. For example, the estimated results can be transmitted to and stored in the database unit 200 via the interface unit 110. This allows the estimated results to be referenced later, or used later to update a trained model or generate a new trained model.

[0115] Diseases whose condition can be estimated by the estimation means 142 are typically diseases that present with outward abnormalities of the hands or nails, and include, but are not limited to, acromegaly, Cushing's syndrome, pseudohypoparathyroidism, rheumatoid arthritis, lung cancer, anemia, osteoarthritis, scleroderma, and psoriatic arthritis. In a preferred embodiment, acromegaly may be a disease whose condition can be estimated by the estimation means 142.

[0116] The estimation means 142 can estimate the subject's disease state using any algorithm. The estimation means 124 can estimate the subject's disease state using, for example, a pre-trained model. The pre-trained model is a model that has learned the relationship between the subject's hand image and the subject's disease state.

[0117] In one example, the trained model may be a model created by the processor unit 120 or processor unit 120' described above, or by the process 500 shown in Figure 5A or the process 510 shown in Figure 5B.

[0118] For example, a trained model has learned the relationship between hand images taken in a specific hand state and orientation, and the state of a disease depicted in the hand image. Therefore, when a hand image taken in a specific hand state and orientation is input, it can output the state of a disease depicted in the hand image. The output could be, for example, a score indicating the presence of a specific disease, or a score indicating the absence of that disease, or both. Alternatively, the output could be, for example, a score indicating the level of a specific disease.

[0119] Figure 3D shows an example of the configuration of a processor unit 140', which is an alternative embodiment of the processor unit 140. The processor unit 140' may have the same configuration as the processor unit 140, except that it includes processing means 143. The system 100 may have a processor unit 140' instead of a processor unit 140. In Figure 3D, components having the same configuration as those described above with reference to Figure 3C are given the same reference numerals, and detailed explanations are omitted here.

[0120] The receiving means 141 is configured to receive target data. The target data received by the receiving means 141 is provided to the processing means 143.

[0121] The processing means 143 is configured to perform preprocessing to remove personal information from the hand images included in the input subject data.

[0122] Figure 4B schematically illustrates an example of a flow 410 in which personal information is removed from a hand image by the processing means 143.

[0123] The processing means 143 is provided with images of the hands of multiple subjects from the receiving means 141. The processing means 143 performs the following steps for each image of the subject's hand.

[0124] In step S411, the processing means 143 detects a first region and a second region in the image of each subject's hand. The first region is a region corresponding to a personal information portion containing the subject's personal information. The personal information portion is, for example, a fingerprint, but is not limited to that. The second region is a region corresponding to an important information portion that is important for estimating the state of a disease. The important information portion may be, for example, a predetermined part where the presence or degree of a disease makes a significant difference in appearance. The important information portion may be, for example, the fingertips and nails, or other fingertips, when estimating a condition related to acromegaly, but is not limited to that. As an example, the important information portion may be a part of the hand that has a significant impact on the estimation result of the second region detection machine learning model according to the present invention, and such a part of the hand may be, for example, the joints of the fingers and / or the area around the thenar eminence. The detection of the first and second regions by the processing means 143 can be performed using methods that are publicly known or may become publicly known in the art.

[0125] For example, the detection of the first region in the hand image can be performed using the same processing as described above for the preprocessing means 123.

[0126] For example, the detection of a second region in a hand image can be performed by the same processing as described above for the preprocessing means 123. For example, the second region detection machine learning model used by the processing means 143 may be a trained machine learning model created by the learning system 11, the second region detection machine learning model used by the preprocessing means 123, or a completely different machine learning model. As an example, the second region detection machine learning model may be a machine learning model provided from outside the system 100.

[0127] In step S412, the processing means 143 determines whether the first region and the second region overlap. Whether or not they overlap may be determined by whether or not at least a part of the first region and the second region overlap, or by using a threshold value that is set as appropriate. For example, whether or not they overlap may be determined by whether or not the ratio of the area of ​​the overlapping region to the sum of the areas of the first region and the second region exceeds a certain threshold value.

[0128] If the first region and the second region do not overlap (for example, in the upper center image of Figure 4B), in step S413, the processing means 143 masks the first region. Masking is performed, for example, by filling in at least a portion of the first region with a specific color. In the example shown in Figure 4B, the portion of the first region corresponding to the fingerprint is filled in black.

[0129] If the first region and the second region overlap (for example, in the case of the image in the lower center of Figure 4B), in step S414, the processing means 143 removes that image from the subject data.

[0130] As shown in Figure 4B, the processing means 143 can remove personal information from the hand image.

[0131] Although Figures 3D and 4B illustrate the processing steps S411 to S414 performed by the processing means 143, the present invention is not limited thereto. The processing steps S411 to S414 may, for example, be performed by the terminal device 300 before the subject data is transmitted to the system 100. This allows the system 100 to estimate the disease state without receiving images containing personal information, further reducing the risk of data management.

[0132] Referring again to Figure 3D, the subject data preprocessed by the processing means 143 is provided to the estimation means 142.

[0133] The estimation means 142 is configured to estimate the disease status of the subject based on a pre-processed image of the subject's hand.

[0134] Each component of the system 100 described above may consist of a single hardware component or multiple hardware components. If it consists of multiple hardware components, the manner in which each hardware component is connected is not limited. Each hardware component may be connected wirelessly or via a wired connection. The system 100 of the present invention is not limited to a specific hardware configuration. Configuring the processor units 120, 120', 140, and 140' using analog circuits instead of digital circuits is also within the scope of the present invention. The configuration of the system 100 of the present invention is not limited to those described above, as long as its function can be realized.

[0135] 3. System Processing for Estimating Disease-Related Conditions Figure 5A shows a flowchart illustrating an example of processing in system 100. Processing 500 is the process for creating a machine learning model to estimate disease-related conditions. Processing 500 is executed by the processor unit 120 of system 100.

[0136] In step S501, the receiving means 121 of the processor unit 120 receives the training data.

[0137] The training data includes images of the hands of multiple subjects, and each subject's hand image is labeled with the subject's disease status (e.g., normal, or mild, moderate, or severe disease status). For example, the disease status label may be "0" indicating no disease or normal status, "1" indicating disease present or abnormal status, or a numerical value between 0 and 1 depending on the severity of the disease. The hand images are images obtained by photographing a hand in a predetermined state from a predetermined direction. The hand images may be labeled by a predetermined state and predetermined direction. The hand images may all be images obtained by photographing a single hand in a single predetermined state from a single predetermined direction, or they may include images obtained by photographing hands in different predetermined states from different predetermined directions. The predetermined state and predetermined direction may be any one that corresponds to the diagnosis of the disease. Preferably, the predetermined state is a hand condition in which a noticeable difference in the appearance of the hand or nails is apparent depending on the presence or degree of disease. Specifically, the predetermined state of the hand may be a hand with at least one finger bent, a hand with all fingers bent, a hand in a fist, or a hand with all fingers extended. Preferably, the predetermined direction for photography is a direction in which a noticeable difference in the appearance of the hand or nails is clearly visible in the image. Specifically, the predetermined direction may be approximately perpendicular to the palm or approximately perpendicular to the back of the hand. In one example, the hand image includes an image of a hand with at least one finger bent, taken from the palm side, preferably a hand with at least one finger bent, which may be a fist. In one example, the hand image includes an image of a hand with all fingers extended, taken from the back of the hand. In one example, the hand image includes an image of a hand with at least one finger bent, taken from the palm side, as well as an image of a hand taken from the back of the hand or an image of a hand with all fingers extended, taken from the back of the hand. The predetermined hand position and shooting direction can be determined based on the principle that the subject's personal information (e.g., fingerprints) does not appear in the image.For example, a clenched fist is a hand position where fingerprints from at least the second to fifth fingers do not appear in the image. For example, a hand with all fingers extended is a hand position where fingerprints from all fingers do not appear in the image when photographed from the back of the hand.

[0138] In one example, to create a machine learning model capable of estimating the condition of acromegaly, images of the hands of multiple subjects may include images of the hands of subjects with acromegaly and images of the hands of subjects without acromegaly. Images of the hands of subjects with acromegaly may be labeled, for example, as positive or as indicating the severity of the acromegaly disease. Images of the hands of subjects without acromegaly may be labeled, for example, as negative.

[0139] For example, to create a machine learning model capable of estimating the state of multiple diseases, images of the hands of multiple subjects may include, for each of the multiple diseases, images of the hands of subjects with the disease and images of the hands of subjects without the disease. For example, to create a machine learning model capable of estimating the state of two diseases, images of the hands of multiple subjects may include images of the hands of subjects with a first disease, images of the hands of subjects without the first disease, images of the hands of subjects with a second disease, and images of the hands of subjects without the second disease. Images of the hands of subjects with the first disease may be labeled, for example, with a label indicating positivity for the first disease or a label indicating the severity of the first disease. Images of the hands of subjects without the second disease may be labeled, for example, with a label indicating negativity for the second disease. Images of the hands of subjects with the second disease may be labeled, for example, with a label indicating positivity for the second disease or a label indicating the severity of the second disease. Images of the hands of subjects without the second disease may be labeled, for example, with a label indicating negativity for the second disease.

[0140] In step S502, the creation means 122 of the processor unit 120 creates a trained machine learning model by training a pre-training machine learning model with the relationship between the hand images of each subject received in step S501 and the disease status of each subject. Training the pre-training machine learning model with the relationship between the hand images of each subject and the disease status of each subject can be done using methods that are publicly known or may become publicly known in the art.

[0141] For example, a machine learning model can be a neural network model, and more specifically, a convolutional neural network (CNN). In one example, the machine learning model could be a ResNet-based model.

[0142] For example, when training a neural network model with training data, an image of a hand may be applied to the input layer, and a label indicating the disease state attached to that image may be applied to the output layer. That is, a pair of input data for training and output data may be (image of a hand, label indicating the disease state attached to the hand image), for example, (image of the hand of the first subject, label indicating the disease state of the first subject), (image of the hand of the second subject, label indicating the disease state of the second subject), ... (image of the hand of the nth subject, label indicating the disease state of the nth subject).

[0143] In step S502, the creation means 122 may create a trained machine learning model by fine-tuning an already created machine learning model. This is preferable because it allows for the creation of a trained machine learning model tailored to a specific group of subjects.

[0144] The process 500 described above creates a machine learning model that estimates the state of a disease. The trained machine learning model thus created has learned the relationship between each subject's hand image and their state of disease for multiple subjects, so when a subject's hand image is input, it can output the subject's state of disease. The output could be, for example, a score indicating the probability of having a particular disease. Alternatively, the output could be, for example, a score indicating the level of a particular disease.

[0145] In step S502, the creation means 122 may create a trained machine learning model by training a pre-trained machine learning model on the relationship between an image of each subject's hand, a predetermined state of the hand, a predetermined direction for photography, and the disease state of each subject. For example, when training a neural network model with training data, an image of the hand and labels indicating a predetermined state and direction may be applied to the input layer, and a label indicating the disease state attached to the image may be applied to the output layer. That is, a pair of input data for training and output data may be (an image of the hand and labels indicating a predetermined state and direction attached to the image of the hand, and a label indicating the disease state attached to the image of the hand), for example, (an image of the first subject's hand and labels indicating a predetermined state and direction attached to the image of the hand, and a label indicating the disease state of the first subject), (an image of the second subject and labels indicating a predetermined state and direction attached to the image of the hand, and a label indicating the disease state of the second subject), ... (an image of the nth subject and labels indicating a predetermined state and direction attached to the image of the hand, and a label indicating the disease state of the nth subject). The trained machine learning model created in this way has learned the relationship between images of each subject's hand, a predetermined hand position, a predetermined orientation for photography, and the disease status of each subject for multiple subjects. Therefore, when images of a subject's hand, a predetermined hand position, and a predetermined orientation for photography are input, the model can output the disease status of that subject.

[0146] In step S502, the creation means 122 may create a trained machine learning model by training a pre-trained machine learning model on the relationship between each subject's hand image and each subject's status related to multiple diseases. The trained machine learning model thus created has learned the relationship between each subject's hand image and each subject's status related to multiple diseases for multiple subjects, so when a subject's hand image is input, it can output each of the subject's status related to multiple diseases. The output may be, for example, a score indicating the probability of each of the multiple diseases being present. Alternatively, the output may be, for example, a score indicating the level of each of the multiple diseases.

[0147] In step S502, the creation means 122 may create a trained machine learning model by training a pre-training machine learning model on the relationship between multiple types of hand images of each subject and the disease state of each subject. The multiple types of hand images include, for example, an image of a hand with at least one finger bent, taken from the palm side, and an image of a hand taken from the back side, and more preferably, an image of a clenched fist, taken from the palm side, and an image of an open hand, taken from the back side.

[0148] Figure 5B shows a flowchart illustrating another example of processing in system 100. Process 510 is a process for creating a machine learning model to estimate disease-related conditions. Process 510 is executed by the processor unit 120' of system 100.

[0149] In step S511, the receiving means 121 of the processor unit 120' receives the training data. Step S511 is the same as step S501.

[0150] In steps S512 to S515, the preprocessing means 123 of the processor unit 120' performs preprocessing on the images of the hands of multiple subjects provided by the receiving means 121.

[0151] In step S512, the preprocessing means 123 detects a first region and a second region in the image of each subject's hand. The first region corresponds to a personal information portion containing the subject's personal information. The personal information portion is, for example, a fingerprint, but is not limited thereto. The second region corresponds to a critical information portion that is important for estimating the state of a disease. The critical information portion may be, for example, a predetermined part where the presence or degree of a disease makes a significant difference in appearance. The critical information portion is, for example, the fingertips, nails, and other fingertips, but is not limited thereto when creating a machine learning model capable of estimating the state of acromegaly. As an example, the critical information portion may be a part of the hand that has a significant impact on the estimation results of the second region detection machine learning model according to the present invention, and such a part of the hand may be, for example, the joints of the fingers and / or the area around the thenar eminence. Detection of the first and second regions in the hand image can be performed using methods that are publicly known or may become publicly known in the art.

[0152] For example, the preprocessing means 123 can detect a first region by performing a region extraction process to extract the hand region in the image and a skeletal extraction process to extract the hand skeleton.

[0153] For example, the preprocessing means 123 can detect a second region by utilizing a second region detection machine learning model that has been trained to detect the second region. The preprocessing means 123 may detect the second region directly from the output of the second region detection machine learning model, or it may detect the second region based on the magnitude of its influence on the output of the second region detection machine learning model.

[0154] In step S513, the preprocessing means 123 determines whether the first region and the second region overlap. Whether or not they overlap may be determined by whether or not at least a part of the first region and the second region overlap, or by using a threshold value that is set as appropriate. For example, whether or not they overlap may be determined by whether or not the ratio of the area of ​​the overlapping region to the sum of the areas of the first region and the second region exceeds a certain threshold value.

[0155] If the first region and the second region overlap, in step S514, the preprocessing means 123 removes the image from the training data. If personal information is masked as in step S515 described later, important information is also masked, making it difficult to estimate diseases with high accuracy. Even if an image with such important information masked is used for training, it is unlikely that an estimate with high accuracy can be expected.

[0156] If the first region and the second region do not overlap, in step S515, the preprocessing means 123 masks the first region. Masking is performed, for example, by filling in at least a portion of the first region with a specific color. The processing in steps S512 to S515 may be performed by the terminal device 300, for example, before the training data is received by the system 100. This will allow the system 100 to create a machine learning model that estimates disease-related conditions without receiving images containing personal information, further reducing data management risks.

[0157] In step S516, the creation means 122 of the processor unit 120' creates a trained machine learning model by training the pre-trained machine learning model on the relationship between the pre-processed images of each subject's hands and the disease state of each subject. Step S516 is the same as step S502.

[0158] The trained model created by the processes 500 and 510 described above can be used, for example, in the processor unit 140 or the processor unit 140'. The parameters of the trained model thus created can be stored in the database unit 200. For example, the created trained model or its parameters can be provided to an external party. For example, the created trained model or its parameters can be provided to another computer via a network N. In this case, the system 100 can establish communication with the other computer, encrypt the created trained model or its parameters, and then transmit the digital data of the created trained model or its parameters, along with a decryption key, to the other computer.

[0159] Figure 6A shows a flowchart illustrating an example of processing in system 100. Process 600 is a process for estimating the state related to a disease. Process 600 is executed by the processor unit 140 of system 100.

[0160] In step S601, the receiving means 141 of the processor unit 140 receives the target data.

[0161] The subject data includes images of the subject's hands. The hand images are obtained by photographing a hand in a predetermined state from a predetermined direction. The predetermined state and direction can be any as appropriate depending on the diagnosis of the disease. Preferably, the predetermined state is a hand condition in which a significant difference in the appearance of the hand or nails is evident depending on the presence or degree of the disease. Specifically, the predetermined state of the hand may be a hand with at least one finger bent, all fingers bent, a hand in a fist, or all fingers extended. Preferably, the predetermined direction for photography is a direction in which a difference in the appearance of the hand or nails is clearly visible in the image. Specifically, the predetermined direction may be approximately perpendicular to the palm or approximately perpendicular to the back of the hand. In one example, the hand image includes an image of a hand with at least one finger bent, photographed from the palm side. Preferably, the hand with at least one finger bent may be a hand in a fist. In another example, the hand image includes an image of a hand with all fingers extended, photographed from the back of the hand. In one example, the hand images include an image of a hand with at least one finger bent, taken from the palm side, as well as an image of a hand taken from the back side, or an image of a hand with all fingers extended, taken from the back side. The hand images may be further labeled by a predetermined hand state and / or predetermined shooting direction. The predetermined hand state and predetermined shooting direction may be determined on the basis that the subject's personal information (e.g., fingerprints) does not appear in the image. For example, a clenched fist is a hand state in which at least the fingerprints of the second to fifth fingers do not appear in the image. For example, all fingers extended is a hand state in which, when photographed from the back side, the fingerprints of all fingers do not appear in the image.

[0162] Preferably, the predetermined state and direction in the subject data are the same as the predetermined state and direction in the training data. This enables more accurate estimation by the estimation means 142 described later.

[0163] In step S602, the estimation means 142 of the processor unit 140 estimates the subject's disease condition based on an image of the subject's hand.

[0164] In step S602, the estimation means 142 can estimate, for example, whether the subject has a certain disease or not, as a state relating to a certain disease. Alternatively, in addition to or instead of the above, the estimation means 142 can estimate the level of a certain disease the subject has (e.g., mild, moderate, severe), as a state relating to a certain disease. Alternatively, in addition to or instead of the above, the estimation means 142 can estimate which of several diseases the subject has, as a state relating to multiple diseases. Alternatively, in addition to or instead of the above, the estimation means 142 can estimate the state relating to a disease based on images of multiple hands of the subject. Alternatively, the estimation means 142 can estimate the state relating to a disease based on images of multiple types of hands of the subject. Multiple types of hand images include, for example, an image of a hand with at least one finger bent, taken from the palm side, and an image of a hand taken from the back side, and more preferably, an image of a clenched fist, taken from the palm side, and an image of an open hand, taken from the back side.

[0165] The results estimated by the estimation means 142 can be output to the outside of the system 100 via the interface unit 110. For example, the estimated results can be transmitted to the subject's terminal device 300 via the interface unit 110. This allows the subject to check their own status via their terminal device 300. For example, the estimated results can be transmitted to the doctor's terminal device 300 via the interface unit 110. This allows the doctor to use the estimated results to help diagnose the subject. For example, in this case, the system 100 can establish communication with the subject's or doctor's terminal device 300, encrypt the estimated results, and transmit the digital data of the estimated results to the terminal device 300 along with a key for decryption. For example, the estimated results can be transmitted to and stored in the database unit 200 via the interface unit 110. This allows the estimated results to be referenced later, or used later to update a trained model or generate a new trained model.

[0166] If the estimation means 142 estimates the presence of a disease, additional data or items may be provided to the subject or physician in addition to the estimated result. The additional data may be, for example, data showing advice for treating the disease, or data showing a treatment plan for treating the disease. The additional items may be drugs, foods, or beverages effective for treating the disease. Even if the estimation means 142 estimates the absence of a disease, additional data or items may be provided to the subject or physician in addition to the estimated result. The additional data may be, for example, data showing advice for preventing the disease. The additional items may be drugs, foods, or beverages effective for preventing the disease. The additional data or items may be derived, for example, using LLM.

[0167] Diseases whose condition can be estimated by the estimation means 142 are typically diseases that present with outward abnormalities of the hands or nails, and include, but are not limited to, acromegaly, Cushing's syndrome, pseudohypoparathyroidism, rheumatoid arthritis, lung cancer, anemia, osteoarthritis, scleroderma, and psoriatic arthritis. In a preferred embodiment, acromegaly may be a disease whose condition can be estimated by the estimation means 142.

[0168] The estimation means 142 can estimate the subject's disease state using any algorithm. The estimation means 124 can estimate the subject's disease state using, for example, a pre-trained model. The pre-trained model is a model that has learned the relationship between the subject's hand image and the subject's disease state.

[0169] In one example, the trained model may be a model created by the processor unit 120 or processor unit 120' described above, or by the process 500 shown in Figure 5A or the process 510 shown in Figure 5B.

[0170] For example, a trained model has learned the relationship between hand images taken in a specific hand state and orientation, and the state of a disease depicted in the hand image. Therefore, when a hand image taken in a specific hand state and orientation is input, it can output the state of a disease depicted in the hand image. The output could be, for example, a score indicating the presence of a specific disease, or a score indicating the absence of that disease, or both. Alternatively, the output could be, for example, a score indicating the level of a specific disease.

[0171] Figure 6B shows a flowchart illustrating another example of processing in system 100. Process 610 is a process for estimating the state related to a disease. Process 610 is executed by the processor unit 140' of system 100.

[0172] In step S611, the receiving means 141 of the processor unit 140' receives the subject data. Step S611 is the same as step S601.

[0173] In steps S612 to S615, the processing means 143 of the processor unit 140' performs preprocessing on the image of the subject's hand provided by the receiving means 141.

[0174] In step S612, the processing means 143 detects a first region and a second region in the image of the subject's hand. The first region corresponds to a personal information portion containing the subject's personal information. The personal information portion is, for example, a fingerprint, but is not limited to that. The second region corresponds to an important information portion that is important for estimating the state of a disease. The important information portion is, for example, a predetermined part where the presence or absence or degree of a disease clearly shows a difference in appearance. The important information portion is, for example, the fingertips and nails, etc., when estimating a condition related to acromegaly, but is not limited to that. As an example, the important information portion may be a part of the hand that has a large influence on the estimation result of the second region detection machine learning model according to the present invention, and such a part of the hand may be, for example, the joints of the fingers and / or the area around the thenar eminence. Detection of the first region and the second region in the hand image can be performed using methods that are publicly known or may become publicly known in the art.

[0175] For example, the processing means 143 can detect a first region by performing a region extraction process to extract the hand region from the image and a skeletal extraction process to extract the hand skeleton.

[0176] For example, the processing means 143 can detect a second region by utilizing a second region detection machine learning model that has been trained to detect the second region. The processing means 143 may detect the second region directly from the output of the second region detection machine learning model, or it may detect the second region based on the magnitude of its influence on the output of the second region detection machine learning model.

[0177] In step S613, the processing means 143 determines whether the first region and the second region overlap. Whether or not they overlap may be determined by whether or not at least a part of the first region and the second region overlap, or by using a threshold value that is set as appropriate. For example, whether or not they overlap may be determined by whether or not the ratio of the area of ​​the overlapping region to the sum of the areas of the first region and the second region exceeds a certain threshold value.

[0178] If the first region and the second region overlap, in step S614, the processing means 143 removes the image from the training data. If personal information is masked as in step S615 described later, important information is also masked, making it difficult to estimate diseases with high accuracy. Even if an image with such important information masked is used for training, it is unlikely that an estimate with high accuracy can be expected.

[0179] If the first region and the second region do not overlap, in step S615, the processing means 143 masks the first region. Masking is performed, for example, by filling in at least a portion of the first region with a specific color.

[0180] The processing in S612 to S615 may be performed by the terminal device 300, for example, before the subject data is received by the system 100. This allows the system 100 to estimate the disease status without receiving images containing personal information, further reducing the risk to data management.

[0181] In step S616, the estimation means 142 of the processor unit 140' estimates the subject's disease condition based on an image of the subject's hand. Step S616 is the same as step S602.

[0182] The disease status of the subject estimated by the processes 600 and 610 described above may be output to the outside of the system 100 via the interface unit 110, or it may be stored in the database unit 200. The disease status of the subject estimated by the processes 600 and 610 described above may be provided to the outside of the system 100 together with additional data or additional items, as described above.

[0183] (Example 1) A trained model was constructed using hand images of subjects with and without acromegaly. Furthermore, the performance of the constructed trained model was evaluated.

[0184] This embodiment was carried out by performing the following steps using a Python script that performs an image classification task using a deep learning model. The following libraries, mainly related to PyTorch, were used to implement the following steps: ・pathlib, os, pandas, numpy: Basic libraries for general file operations and data processing. ・skimage, PIL: Image processing libraries for image loading and preprocessing. ・torch, torchvision: PyTorch-related libraries for model building, data processing, and managing the learning process. ・sklearn.metrics: Evaluation library for calculating ROC curves and AUC scores.

[0185] (Preparation of Dataset) A dataset consisting of 1,306 hand images from subjects with acromegaly and 3,744 hand images from subjects without acromegaly was used. Each image is either a clenched fist photographed from the palm side (hereinafter referred to as a clenched fist image) or a hand with all fingers extended photographed from the back side (hereinafter referred to as a back-of-hand image).

[0186] The above dataset was subjected to resizing, random cropping, horizontal flipping, and normalization. This augments the data and improves the generalization performance of the model.

[0187] The above datasets were randomly divided into three sets: a training dataset, a validation and adjustment dataset, and an evaluation dataset. The ratio of the training dataset, validation and adjustment dataset, and evaluation dataset was set to 8:1:1.

[0188] (Model Creation) The training dataset and the validation and tuning datasets were used as training data, and the ResNet50 was used as the base model to create the model. Models were created under the following three different conditions: Condition A. The model was created by using fist images and back-of-hand images as training data and training the model with optimized hyperparameters. Condition B1. The model was created by using fist images and back-of-hand images as training data and training the model with appropriate hyperparameters. Condition B2. The model was created by using only fist images as training data and training the model with the same appropriate hyperparameters as in B1.

[0189] (Model Evaluation) For the created model, the AUC score was calculated on the evaluation dataset.

[0190] For the model under condition A, a diagnostic accuracy of approximately 0.92 AUC score was obtained. Therefore, this model demonstrates excellent diagnostic performance for acromegaly.

[0191] For the model under condition B1, an average improvement of approximately 0.02 in the AUC score was observed compared to condition B2. Therefore, it was demonstrated that hand images are effective in improving the diagnostic accuracy of this model.

[0192] (Example 2) Using hand images obtained from case groups 1-6, each exhibiting different hand abnormalities, and a control group, a trained model was constructed, and the performance of the constructed trained model was evaluated.

[0193] (Preparation of Datasets) The datasets for the following case groups 1-6 and the control group were used. Each image is either a clenched fist or a back of the hand.

[0194] Case Group 1: Hand images of subjects with acromegaly under the age of 57. Case Group 2: Hand images of subjects with acromegaly 57 years of age or older. Case Group 3: Hand images of subjects with acromegaly whose symptoms have gone into remission. Case Group 4: Hand images of subjects with acromegaly whose symptoms have not gone into remission. Case Group 5: Hand images of male subjects with acromegaly. Case Group 6: Hand images of female subjects with acromegaly. Control Group: Hand images of subjects without acromegaly.

[0195] Each of the above datasets was subjected to resizing, random cropping, horizontal flipping, and normalization. Each of the above datasets was randomly classified into three datasets: a training dataset, a validation and adjustment dataset, and an evaluation dataset. The ratio of the training dataset, validation and adjustment dataset, and evaluation dataset was set to 8:1:1.

[0196] (Model Creation) Using ResNet50 as the base model, the following models 1 to 6 were created. Model 1: The training dataset and validation and adjustment datasets of case group 1 and the control group were used as training data. Model 2: The training dataset and validation and adjustment datasets of case group 2 and the control group were used as training data. Model 3: The training dataset and validation and adjustment datasets of case group 3 and the control group were used as training data. Model 4: The training dataset and validation and adjustment datasets of case group 4 and the control group were used as training data. Model 5: The training dataset and validation and adjustment datasets of case group 5 and the control group were used as training data. Model 6: The training dataset and validation and adjustment datasets of case group 6 and the control group were used as training data.

[0197] (Model Evaluation) The following evaluations were performed on each created model. The evaluation indicators used were ROC-AUC score, sensitivity, specificity, PPV, NPV, and F1 score. Evaluation 1: Model 1 was evaluated on the evaluation dataset of case group 1 and the control group. Evaluation 2: Model 2 was evaluated on the evaluation dataset of case group 2 and the control group. Evaluation 3: Model 3 was evaluated on the evaluation dataset of case group 3 and the control group. Evaluation 4: Model 4 was evaluated on the evaluation dataset of case group 4 and the control group. Evaluation 5: Model 5 was evaluated on the evaluation dataset of case group 5 and the control group. Evaluation 6: Model 6 was evaluated on the evaluation dataset of case group 6 and the control group.

[0198] (Results) The results for evaluations 1 to 6 are shown in Table 1.

[0199] For Evaluation 1, a diagnostic accuracy of approximately 0.96 AUC score was obtained. Model 1 demonstrated excellent diagnostic performance for acromegaly patients under 57 years of age.

[0200] Regarding evaluation 2, a diagnostic accuracy of approximately 0.95 AUC score was achieved. Model 2 demonstrated excellent diagnostic performance for acromegaly patients aged 57 years or older.

[0201] For evaluation 3, a diagnostic accuracy of approximately 0.94 AUC score was obtained. Model 3 demonstrated excellent diagnostic performance in subjects whose acromegaly symptoms were in remission.

[0202] For evaluation 4, a diagnostic accuracy of approximately 0.97 AUC score was obtained. Model 4 demonstrated excellent diagnostic performance in subjects whose acromegaly symptoms were not in remission.

[0203] For evaluation 5, a diagnostic accuracy of approximately 0.97 AUC score was achieved. Model 5 demonstrated excellent diagnostic performance for male patients with acromegaly.

[0204] For evaluation 6, a diagnostic accuracy of approximately 0.94 AUC score was achieved. Model 6 demonstrated excellent diagnostic performance for female patients with acromegaly.

[0205] Furthermore, regarding evaluations 1 and 2, patients in case group 1 exhibit different external hand abnormalities than patients in case group 2. For example, the hands of acromegaly patients under 57 years of age exhibit clear external abnormalities in the fingertips, nails, and other fingertips, while the hands of acromegaly patients 57 years of age or older may exhibit indistinct external abnormalities due to the aging of the hands.

[0206] Furthermore, regarding evaluations 3 and 4, patients in case group 3 exhibit different hand appearance abnormalities than patients in case group 4. For example, the hands of subjects whose acromegaly symptoms have not gone into remission exhibit appearance abnormalities, including changes in the shape of the fingertips due to changes in the shape of the finger bones (e.g., cauliflower-like changes) and changes in the overall shape of the hand due to hypertrophy of the muscles, fat, and connective tissue of the hand. On the other hand, the hands of subjects whose acromegaly symptoms have gone into remission retain changes in the shape of the fingertips and hypertrophy of the phalanges due to changes in the shape of the finger bones, but the overall hypertrophy of the hand due to hypertrophy of muscles, fat, and connective tissue is partially resolved due to remission. This is because the changes in the shape of the bones are not resolved.

[0207] Furthermore, regarding evaluations 5 and 6, patients in case group 5 exhibit different hand abnormalities than patients in case group 6. For example, the average ratio of thickness between the enlarged portion and the non-enlarged portion may differ between male patients with acromegaly and female patients with acromegaly.

[0208] As described above, the trained model constructed according to the present invention demonstrated excellent diagnostic performance for case groups 1 to 6, each exhibiting different hand appearance abnormalities. The present invention is not limited to specific diseases that present with specific hand appearance abnormalities, but can be applied to the diagnosis of various diseases that present with various hand appearance abnormalities. For example, the present invention can be applied to the diagnosis of diseases such as rheumatoid arthritis or osteoarthritis presenting with joint deformities, lung cancer presenting with clubbing of the fingers, iron deficiency anemia presenting with spoon nails, Cushing's syndrome presenting with pigmentation or thinning of the skin, scleroderma presenting with discoloration of the fingertips, psoriatic arthritis presenting with joint swelling, and pseudohypoparathyroidism presenting with brachydactyly.

[0209] This disclosure is useful as it provides a system for estimating disease-related conditions or a system for creating machine learning models for estimating disease-related conditions.

[0210] 11 Learning system 12 Estimation system 100 System 110 Interface unit 120, 120', 140, 140' Processor unit 130 Memory unit 200 Database unit 300 Terminal device N Network

Claims

1. A system for creating a machine learning model for estimating disease-related conditions, the system comprising: a receiving means for receiving training data, the training data comprising images of the hands of each of a plurality of subjects, the images of the hands of the subjects being labeled with the disease-related conditions of the subjects, and the images comprising images of hands taken from the palm side with at least one finger bent; and a creation means for creating a trained machine learning model by training using the training data.

2. The system further comprises preprocessing means for preprocessing the training data, wherein the preprocessing includes detecting a first region and a second region in the image of each of the plurality of subjects, wherein the first region is a region corresponding to the personal information portion of the subject's hand and the second region is a region corresponding to the important information portion of the subject's hand; determining whether the first region and the second region overlap; removing the image from the training data if the first region and the second region overlap; and masking the first region if the first region and the second region do not overlap, and the creation means performs a training process using the preprocessed training data, the system according to claim 1.

3. The system according to claim 2, wherein detecting the first region includes extracting a region of a hand in the image, extracting the skeleton of the hand in the extracted region, and detecting the first region based on the extracted skeleton.

4. The system according to claim 3, wherein the personal information portion includes a fingerprint.

5. The system according to claim 2, wherein detecting the second region includes inputting the image into a second region detection machine learning model and detecting the second region based on the output from the second region detection machine learning model.

6. The system according to claim 5, wherein the second region detection machine learning model learns the relationship between pre-trained images of a hand and the state of the hand, and is configured to output a state estimation result from an input image, and detecting the second region based on the output from the second region detection machine learning model includes identifying a region in the image that at least influenced the estimation result, and detecting the second region based on the identified region.

7. The system according to claim 5, wherein the important information portion includes the fingertip.

8. The system according to claim 1, wherein the hand with at least one bent finger is a fist.

9. The system according to claim 1, further comprising an image taken from the back of the hand.

10. The system according to claim 1, wherein the disease is acromegaly.

11. An estimation system for estimating the disease state of a subject, the estimation system comprising: a receiving means for receiving subject data, the subject data including an image of the subject's hand, the image including an image of the hand taken from the palm side with at least one finger bent; and an estimation means for estimating the subject's state based on the subject data.

12. The estimation system according to claim 11, wherein the estimation means estimates the state using a machine learning model, and the machine learning model is a trained machine learning model created by the system described in any one of claims 1 to 8.

13. The estimation system further comprises processing means for preprocessing the subject data, the estimation system comprising: detecting a first region and a second region in the image, wherein the first region is a region corresponding to the personal information portion of the subject's hand, and the second region is a region corresponding to the important information portion of the subject's hand; determining whether the first region and the second region overlap; removing the image from the subject data if the first region and the second region overlap; and masking the first region if the first region and the second region do not overlap, the estimation system according to claim 11, wherein the estimation means estimates the state based on the preprocessed subject data.

14. The estimation system according to claim 11, wherein the hand with at least one bent finger is a clenched fist.

15. The estimation system according to claim 11, further comprising an image of a hand with all fingers extended, taken from the back of the hand.

16. The estimation system according to claim 11, wherein the disease is acromegaly.

17. A method for creating a machine learning model for estimating disease-related conditions, comprising: receiving training data, wherein the training data includes images of the hands of a plurality of subjects, the images of the subjects' hands are labeled with the subjects' disease-related conditions, and the images include images of hands taken from the palm side with at least one finger bent; and creating a trained machine learning model by training using the training data.

18. A method for estimating the disease state of a subject, comprising: receiving subject data, wherein the subject data includes an image of the subject's hand, the image of which includes an image of the hand taken from the palm side with at least one finger bent; and estimating the subject's state based on the subject data.

19. A program for creating a machine learning model for estimating disease-related conditions, the program being executed on a computer equipped with a processor, the program being configured to receive training data, the training data comprising images of the hands of a plurality of subjects, the images of the subjects' hands being labeled with the subjects' disease-related conditions, the images comprising images of hands taken from the palm side with at least one finger bent, and the program causing the processor to perform a training process using the training data to create a trained machine learning model.

20. A program for estimating the disease state of a subject, the program being executed on a computer equipped with a processor, the program causing the processor to perform a process that includes receiving subject data, the subject data including an image of the subject's hand, the image including an image of the hand taken from the palm side with at least one finger bent, and estimating the subject's state based on the subject data.

21. A system for creating training data for machine learning, comprising: receiving means for receiving images of a plurality of subjects; and preprocessing means for processing the images, wherein the preprocessing means is configured to perform the following processing for each of the plurality of subjects' images: detecting a first region and a second region in the image, wherein the first region is a region corresponding to personal information and the second region is a region corresponding to important information; determining whether the first region and the second region overlap; removing the image from the training data if the first region and the second region overlap; and masking the first region if the first region and the second region do not overlap.

22. The system according to claim 21, wherein the image is an image of a hand, and the personal information portion is a fingerprint.

Images