Computing apparatus for dynamically setting cut-off of neural network model on basis of characteristics of medical environment, control method therefor, and computer program

The computing device dynamically adjusts neural network model cut-off values based on medical environment characteristics, improving diagnostic accuracy and compliance by addressing inconsistencies across different institutions.

WO2026240449A1PCT designated stage Publication Date: 2026-11-19MEDICAL AI CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2026/006086
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2026-05-14
Filing Date
2026-05-14
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Uniform application of fixed cut-off values in neural network models across different medical institutions leads to inconsistent diagnostic outcomes, resulting in excessive false positives or false negatives due to variations in patient population characteristics and clinical environments.

Method used

A computing device dynamically sets the cut-off values of a neural network model based on the characteristics of a medical environment, including factors like institution size, location, clinical division, and equipment, using a method that involves identifying clusters of similar environments and adjusting cut-off values based on operational data and performance indicators.

Benefits of technology

This approach enhances diagnostic accuracy by tailoring cut-off values to specific medical environments, reducing false positives and negatives, and ensuring compliance with regulatory requirements through systematic management of cut-off settings.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2026006086_19112026_PF_FP_ABST
    Figure KR2026006086_19112026_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure provides a computing apparatus for dynamically setting a cut-off of a neural network model on the basis of characteristics of a medical environment, a control method therefor, and a computer program. The control method performed by the computing apparatus including at least one processor, according to an embodiment of the present disclosure, comprises the steps of: acquiring a neural network model trained to output a score corresponding to the state of a patient on the basis of biometric data; determining a cut-off value of the neural network model on the basis of characteristics of a medical environment; and analyzing biometric data of the patient on the basis of the neural network model to which the determined cut-off value is applied, to thereby identify a determination result about the state of the patient.
Need to check novelty before this filing date? Find Prior Art

Description

Computing device, control method, and computer program for dynamically setting the cutoff of a neural network model based on the characteristics of a medical environment

[0001] The present disclosure relates to artificial intelligence technology in the medical field, and more specifically, to a computing device that dynamically sets a cutoff of a neural network model based on the characteristics of a medical environment, a control method thereof, and a computer program.

[0002] With the recent advancement of Artificial Intelligence (AI), technologies that acquire users' biometric data using wearable devices or mobile electronic devices and analyze their condition in real time based on this data are becoming commonplace. For example, this includes technology that analyzes electrocardiogram (ECG) data acquired through a smartwatch using an AI model to identify a user's condition related to heart disease. Through such biometric data analysis technology utilizing AI, users can continuously monitor for abnormal signs or their health status without visiting a medical institution.

[0003] Such artificial intelligence-based biometric data analysis technologies are being actively adopted in medical institutions. AI software implemented in these institutions analyzes biometric data acquired from patients to output diagnostic scores for specific diseases and is utilized to assist medical staff in clinical decision-making by determining positive or negative results based on preset cut-off values. Traditionally, these cut-off values ​​have been applied uniformly as fixed values ​​across all medical institutions. However, if the same cut-off value is applied across the board despite variations in patient population characteristics, prevalence rates, and clinical environments among institutions, it can lead to an excessive number of false positives or false negatives in specific facilities, posing a risk of resulting in clinically inappropriate diagnostic outcomes.

[0004] The present disclosure is conceived in response to the aforementioned background technology and aims to provide a computing device that dynamically sets a cut-off of a neural network model based on the characteristics of a medical environment, a control method thereof, and a computer program.

[0005] However, the problems to be solved in this disclosure are not limited to those mentioned above, and other unmentioned problems may be clearly understood based on the description below.

[0006] According to one embodiment of the present disclosure for solving the problem described above, a method for controlling a computing device that dynamically sets a cutoff of a neural network model based on the characteristics of a medical environment is disclosed, which is performed by a computing device comprising at least one processor. The method may include the steps of: acquiring a neural network model trained to output a score corresponding to a patient's condition based on biological data; determining a cutoff value of the neural network model based on the characteristics of the medical environment; and analyzing the patient's biological data based on the neural network model to which the determined cutoff value is applied to determine a judgment result regarding the patient's condition.

[0007] Alternatively, the step of determining the cutoff value of the neural network model comprises: identifying a cluster corresponding to the medical environment based on at least one of the size of the medical institution, the location of the medical institution, clinical division information, and medical equipment information from which the biometric data was acquired, which are included in the characteristics of the medical environment; and determining the cutoff value based on operational data of the neural network model in at least one other medical environment included in the cluster, wherein the operational data may include a cutoff value applied to the neural network model used in the other medical environment.

[0008] Alternatively, the step of identifying a cluster corresponding to the medical environment may include, for a plurality of different medical environments, identifying a first candidate cluster based on the size of the medical institution; for the first candidate cluster, identifying a second candidate cluster based on the clinical division; for the second candidate cluster, identifying a third candidate cluster based on the location of the medical institution; and for the third candidate cluster, identifying a fourth candidate cluster based on the medical equipment that acquired the biometric data and determining it as a cluster corresponding to the medical environment.

[0009] Alternatively, information regarding the medical equipment that acquired the biometric data includes at least one of the manufacturer of the medical equipment, the model of the medical equipment, and the measurement protocol set on the medical equipment, and the step of identifying the fourth candidate cluster may include identifying the fourth candidate cluster such that, based on the information regarding the medical equipment, the similarity of the distribution of the score output from the neural network model is greater than or equal to a preset value and includes other similar medical environments.

[0010] Alternatively, the method further comprises the steps of: calculating a performance indicator of the neural network model based on reading data of the medical environment accumulated over a preset period after the determined cut-off value is applied to the neural network model; and updating the cut-off value by comparing the calculated performance indicator with a preset target performance indicator, wherein the reading data of the medical environment may include a judgment result of the patient based on the score output from the neural network model and a clinical confirmation result corresponding to the judgment result.

[0011] Alternatively, the step of updating the cutoff value may include, when the calculated performance indicator falls short of the target performance indicator, a step of repeatedly calculating the performance indicator while adjusting the cutoff value stepwise by a preset unit, and a step of updating to a cutoff value that first satisfies the target performance indicator.

[0012] Alternatively, the step of determining the cutoff value of the neural network model may include receiving a target performance indicator for the neural network model from a user, obtaining a Receiver Operating Characteristic (ROC) curve for the neural network model, and determining the cutoff value that satisfies the input target performance indicator based on the ROC curve.

[0013] Alternatively, the step of obtaining an ROC curve for the neural network model may include obtaining an ROC curve corresponding to the cluster based on operational data of the neural network model in at least one other medical environment included in the identified cluster.

[0014] Alternatively, the method may further include the steps of: obtaining anonymized operational data of a plurality of different medical environments included in the identified cluster; calculating distribution information of cutoff values ​​and performance indicators used in the plurality of different medical environments based on the obtained operational data; outputting a user interface that displays the cutoff values ​​determined in the medical environments and the performance indicators of the neural network model together with the distribution information so as to be identifiable; and updating the cutoff values ​​determined in the medical environments based on user input received through the user interface.

[0015] Alternatively, the step of determining the cutoff value of the neural network model comprises: generating a cutoff profile corresponding to the patient based on the patient's classification information; identifying a cutoff profile that matches the cutoff profile corresponding to the patient among a plurality of previously generated cutoff profiles in the medical environment; and determining the cutoff value corresponding to the identified cutoff profile as the cutoff value of the neural network model, wherein the patient's classification information may include at least two of the patient's medical department, purpose of examination, patient age group, and patient symptoms.

[0016] Alternatively, the aforementioned multiple cut-off profiles may include two or more cut-off profiles corresponding to the same medical department, and the two or more cut-off profiles corresponding to the same medical department may be generated to have different cut-off values ​​corresponding to a combination of classification information in which at least one of the examination purpose, the patient age group, and the patient symptoms is different from each other.

[0017] According to another embodiment of the present disclosure for solving the problem described above, a computing device is disclosed for dynamically setting a cutoff of a neural network model based on the characteristics of a medical environment. The computing device includes a memory containing executable program codes and at least one core, and by executing the program codes, it obtains a neural network model trained to output a score corresponding to a patient's condition based on biological data, determines a cutoff value of the neural network model based on the characteristics of the medical environment, and analyzes the patient's biological data based on the neural network model to which the determined cutoff value is applied to determine a judgment result regarding the patient's condition.

[0018] According to another embodiment of the present disclosure for solving the problem described above, a computer program stored on a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, it performs an operation to control a computing device that dynamically sets a cutoff of a neural network model based on the characteristics of a medical environment. The operation may include: acquiring a neural network model trained to output a score corresponding to a patient's condition based on biometric data; determining a cutoff value of said neural network model based on the characteristics of the medical environment; and analyzing the patient's biometric data based on said neural network model to which the determined cutoff value is applied to determine a judgment result regarding the patient's condition.

[0019] The technical solutions obtainable in this disclosure are not limited to the solutions mentioned above, and other solutions not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0020] According to one embodiment of the present invention, by dynamically setting the cutoff value of a neural network model based on the characteristics of the medical environment, it is possible to provide diagnostic results that reflect the characteristics of patient groups according to the medical environment. Through this, false positives or false negatives that may occur due to the uniform application of a single cutoff value can be effectively reduced.

[0021] Furthermore, by objectively determining cutoff values ​​based on operational data from the medical environment, the deterioration of the reliability and consistency of diagnostic results caused by cutoff settings based on the user's subjective judgment can be prevented. Moreover, by systematically managing the process of determining and updating cutoff values, compliance with regulatory requirements for medical devices and reliability in clinical application can be ensured.

[0022] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

[0023] FIG. 1 is a block diagram of a computing device that dynamically sets the cut-off of a neural network model based on the characteristics of a medical environment according to one embodiment of the present disclosure.

[0024] FIG. 2 is a schematic flowchart of a control method for a computing device that dynamically sets a cut-off of a neural network model based on the characteristics of an environment according to one embodiment of the present disclosure.

[0025] FIG. 3 is a flowchart of a control method for a computing device that identifies clusters corresponding to the characteristics of an environment according to one embodiment of the present disclosure and dynamically sets the cut-off of a neural network model.

[0026] FIG. 4 is an exemplary diagram of a control method for a computing device that identifies clusters corresponding to the characteristics of an environment according to one embodiment of the present disclosure and dynamically sets the cut-off of a neural network model.

[0027] FIG. 5 is a flowchart of a method for updating a cutoff value based on a performance indicator of a neural network model according to a determined cutoff value according to one embodiment of the present disclosure.

[0028] FIG. 6 is an exemplary diagram of a method for updating a cutoff value based on a performance indicator of a neural network model according to a determined cutoff value according to one embodiment of the present disclosure.

[0029] FIG. 7 is a flowchart of a method for determining a cut-off value based on an ROC curve according to one embodiment of the present disclosure.

[0030] FIG. 8 is an example diagram of a User Interface (UI) displaying distribution information of cut-off values ​​and performance indicators used in a plurality of different medical environments according to one embodiment of the present disclosure.

[0031] FIG. 9 is a flowchart of a method for determining a cutoff value based on a cutoff profile corresponding to a patient according to one embodiment of the present disclosure.

[0032] FIG. 10 is an example diagram of a cut-off value according to patient classification information according to one embodiment of the present disclosure.

[0033] FIG. 11 is a block diagram of a computing device that dynamically sets the cut-off of a neural network model based on the characteristics of a medical environment according to another embodiment of the present disclosure.

[0034] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art (hereinafter, those skilled in the art) can easily implement them. The embodiments presented in the present disclosure are provided to enable those skilled in the art to use or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be embodied in various different forms and is not limited to the embodiments below.

[0035] Throughout the specification of the present disclosure, identical or similar reference numerals refer to identical or similar components. Additionally, to clearly explain the present disclosure, reference numerals in the drawings that are unrelated to the description of the present disclosure may be omitted.

[0036] The term “or” as used in this disclosure is intended to mean an implicit “or” rather than an exclusive “or.” That is, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” should be understood to mean one of the natural implicit substitutions. For example, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” may be interpreted as X using A, X using B, or X using both A and B.

[0037] The term “and / or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the enumerated related concepts.

[0038] The terms “comprising” and / or “comprising” as used in this disclosure should be understood to mean the presence of certain features and / or components. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, other components and / or combinations thereof.

[0039] Where not otherwise specified in the present disclosure or where it is not clear from the context that the singular form indicates, the singular should generally be interpreted as including “one or more.”

[0040] The term "the N (N is a natural number)" used in this disclosure may be understood as an expression used to distinguish the components of this disclosure from one another according to certain criteria, such as functional perspectives, structural perspectives, or convenience of explanation. For example, components performing different functional roles in this disclosure may be distinguished as a first component or a second component. However, components that are substantially identical within the technical scope of this disclosure but must be distinguished for the convenience of explanation may also be distinguished as a first component or a second component.

[0041] The term “acquisition” as used in this disclosure may be understood to mean not only receiving data through a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

[0042] Meanwhile, the terms "module" or "unit" used in this disclosure may be understood as referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. In this case, "module" or "unit" may be a unit composed of a single element, or a unit expressed as a combination or set of multiple elements. For example, in a narrow sense, "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a procedure implemented through software execution, or a set of instructions for program execution. Furthermore, in a broad sense, "module" or "unit" may refer to the computing device itself that constitutes the system, or an application executed on the computing device. However, since the above-described concept is merely an example, the concepts of "module" or "part" may be defined in various ways within the scope understandable to those skilled in the art based on the contents of this disclosure.

[0043] As used in this disclosure, the term "model" may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model regarding a processing process to solve a specific problem. For example, a neural network "model" may refer to an overall system implemented as a neural network that possesses problem-solving capabilities through learning. In this case, the neural network may possess problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or a set of neural networks composed of multiple neural networks.

[0044] The term "data" as used in this disclosure may include "image," "signal," etc. The term "image" as used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, "image" may be understood as a term referring to a digital representation of an object visible to the human eye. For example, "image" may refer to multidimensional data composed of elements corresponding to pixels in a two-dimensional image. "Image" may refer to multidimensional data composed of elements corresponding to voxels in a three-dimensional image.

[0045] As used in this disclosure, “object” refers to an object or entity to which data is collected or analyzed. For example, in medical data processing, “object” may include a person (patient or subject) to whom biosignal or image data is acquired. Furthermore, in cases involving image processing or object recognition, “object” may be extended to specific objects or structures, and may include biological objects such as animals in addition to people. That is, “object” may encompass various forms of physical or biological objects depending on the purpose of analysis and refers to an entity that acquires data through biosignal measurement devices or sensors, etc.

[0046] The explanation of the foregoing terms is intended to aid in understanding the present disclosure. Accordingly, it should be noted that unless a foregoing term is explicitly stated as a matter limiting the content of the present disclosure, it is not to be used in the sense of limiting the technical concept of the content of the present disclosure.

[0047] FIG. 1 is a block diagram of a computing device that dynamically sets the cut-off of a neural network model based on the characteristics of a medical environment according to one embodiment of the present disclosure.

[0048] Referring to FIG. 1, a computing device (100) according to one embodiment of the present disclosure may include a processor (110), a memory (120), and a communication interface (130). However, since FIG. 1 is merely an example, the computing device (100) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the computing device (100).

[0049] A computing device (100) according to one embodiment of the present disclosure may be a hardware device or a part of a hardware device that performs comprehensive processing and computation of data, or it may be a software-based computing environment connected to a communication network. For example, the computing device (100) may be a server that performs intensive data processing functions and is an entity that shares resources, or it may be a client that shares resources through interaction with the server. Additionally, the computing device (100) may be a cloud system in which a plurality of servers and clients interact to comprehensively process data.

[0050] The description above is merely one example regarding the type of computing device (100), and the type of computing device (100) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure. For example, the computing device (100) may be implemented as various electronic devices such as a desktop, laptop, smartphone, smart watch, cloud server, edge server, etc. Alternatively, the computing device (100) may be implemented as a biosignal measuring device that directly measures biosignals. In addition, the computing device (100) may be implemented as part of a medical information system installed and operated in a medical institution, or in the form of a central server connected to communicate with multiple medical institutions.

[0051] A computing device (100) according to one embodiment of the present disclosure acquires biometric data (10) from a user (1), analyzes the biometric data (10) using a pre-trained neural network model, and can determine the state of the user (1) or predict the possibility of various diseases of the user (1) based on the analysis results. The state of the user (1) may include the user's (1) health status related to a specific disease. For example, the state of the user (1) may include the presence or absence of a specific disease, the possibility of the specific disease manifesting, the severity of the user regarding the specific disease, etc. For example, specific diseases may include various heart-related diseases of the user (Left Ventricular Systolic Dysfunction (LVSD), Left Ventricular Diastolic Dysfunction (LVDD), Myocardial Infarction (MI), Aortic Stenosis (AS), Aortic Regurgitation (AR), etc.).

[0052] In addition, a computing device (100) according to one embodiment of the present disclosure can dynamically determine a cutoff value of a neural network model based on the characteristics of a medical environment to which the neural network model is to be applied, and determine the state of a user (1) by applying the determined cutoff value to the neural network model. Here, the characteristics of the medical environment may include at least one of the size of the medical institution, the location of the medical institution, clinical department information, and medical equipment information that has acquired biometric data. By applying the cutoff value determined based on the characteristics of the medical environment, the computing device (100) can provide a determination result suitable for each medical environment even with the same neural network model. Specific operations related to the dynamic setting of the cutoff value will be described in detail with reference to the drawings described below.

[0053] The biometric data (10) may include various forms of data observed or measured from the user to reflect the user's physiological state. The biometric data (10) may include not only biometric signals collected continuously but also single physiological indicators measured at a specific point in time. For example, the biometric data (10) may include electrocardiogram (ECG) data, electroencephalogram (EEG) data, photoplethysmogram (PPG) data, MRI, CT images, respiration, heart rate, blood pressure, oxygen saturation, body temperature, etc. However, for the convenience of explaining the present disclosure, the biometric data (10) will be described below as electrocardiogram data.

[0054] A computing device (100) can predict the condition of a user (1) regarding diseases, particularly heart disease, by analyzing biometric data (10) based on a pre-trained neural network model. To this end, the neural network model can be pre-trained using multiple biometric data of the same type as input data and the presence or absence of a specific disease as label data. The neural network model can extract feature information latent in the biometric data and learn the correlation between the extracted feature information and the specific disease.

[0055] The computing device (100) can predict the state of a user's disease by analyzing biological data using a neural network model learned in this manner. The computing device (100) can input biological data into the neural network model and determine the state of the user's disease based on the output data (or output value) of the neural network model. In particular, the computing device (100) can input biological data into the neural network model to obtain a score corresponding to the probability of a specific disease. Here, the score is the output data (or output value) of the aforementioned neural network model or a value obtained by applying weights to the output data, and can be interpreted as an indicator representing the probability of a specific disease.

[0056] The computing device (100) can determine the state of a user's specific disease by comparing the acquired score with a value set to distinguish the state of a specific disease. Here, the value set to distinguish the state of a specific disease may be referred to as a "cut-off value" in the present disclosure and may operate as a threshold value for classifying positive / negative, risk group / non-risk group, etc., based on the output score of a neural network model. For example, the computing device (100) can obtain a score corresponding to the possibility of left ventricular systolic dysfunction from a neural network model trained to predict the possibility of left ventricular systolic dysfunction of a user based on electrocardiogram data (10), and determine the user's state (i.e., the presence or absence or severity of left ventricular systolic dysfunction) by comparing the acquired score with a cut-off value (or at least one reference value or reference range) set to classify the risk group of left ventricular systolic dysfunction. At this time, the cut-off value may be determined differently depending on the characteristics of the medical environment, and accordingly, different judgment results may be derived for the same score depending on the medical environment.

[0057] A processor (110) according to one embodiment of the present disclosure may be understood as a constituent unit comprising hardware and / or software for performing computing operations. For example, the processor (110) may read a computer program and perform data processing for machine learning. The processor (110) may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. A processor (110) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). Since the above-described types of processors (110) are merely examples, the types of processors (110) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0058] The processor (110) is connected to other components of the computing device (100) (e.g., memory (120) and communication interface (130)) to control the overall operation of the computing device (100). Additionally, the processor (110) can perform a series of operations to identify characteristics of a medical environment, determine a cutoff value of a neural network model based on the identified characteristics of the medical environment, and analyze the patient's biological data based on the neural network model to which the determined cutoff value is applied to derive a judgment result regarding the patient's condition. Furthermore, the processor (110) may perform an operation to calculate a performance indicator of the neural network model based on the reading data of the medical environment accumulated after the cutoff value is applied, and to update the cutoff value based on the calculated performance indicator.

[0059] A memory (120) according to one embodiment of the present disclosure may be understood as a configuration unit comprising hardware and / or software for storing and managing data processed by a computing device (100). That is, the memory (120) may store data of any form generated or determined by a processor (110) and data of any form received by a communication interface (130) of the computing device (100).

[0060] To this end, the memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. Additionally, the memory (120) may include a database system that controls and manages data in a predetermined system. Since the above-described types of memory (120) are merely examples, the types of memory (120) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0061] The memory (120) can manage data, combinations of data, and program code executable by the processor (110) by structuring and organizing them for the processor (110) to perform operations. For example, the memory (120) may store multiple biometric data received through the communication interface (130), information of an object corresponding to the biometric data, a previously learned neural network model, or a neural network model scheduled for learning.

[0062] Additionally, the memory (120) may store at least one neural network model trained to predict the user's condition regarding a specific disease of the user (1) (and a neural network model scheduled to be trained to predict the user's condition regarding another disease), program code for operating the neural network model to perform training, program code for operating the neural network model to receive biological data input and perform inference according to the purpose of use of the computing device (100), and output data or processed data generated as the program code is executed. Furthermore, the memory (120) may store various data related to the dynamic setting of cut-off values ​​according to one embodiment of the present disclosure. For example, the memory (120) can store characteristic information of the medical environment (e.g., size, location, clinical division, medical equipment information of the medical institution, etc.), cluster identification information corresponding to the medical environment, operational data of the neural network model in another medical environment (e.g., applied cutoff value and performance indicators therefrom, etc.), a cutoff profile generated based on the determined cutoff value and patient classification information, ROC (Receiver Operating Characteristic) curve information of the neural network model, and reading data and clinical confirmation results accumulated after the application of the cutoff value.

[0063] The communication interface (130) can be understood as a configuration unit that transmits and receives data through any known form of wired or wireless communication system. For example, the communication interface (130) can perform data transmission and reception using wired or wireless communication systems such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), 5th generation mobile communication (5G), ultrawide-band wireless communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity (Wi-Fi), near field communication (NFC), or Bluetooth. Since the communication systems described above are merely examples, wired or wireless communication systems for data transmission and reception of the communication interface (130) may be applied in various ways other than those described above. The communication interface (130) can receive data necessary for the processor (110) to perform calculations through wired or wireless communication with any system or any client, etc.

[0064] Additionally, the communication interface (130) can transmit data generated through the computation of the processor (110) via wired or wireless communication with any system or any client. For example, the communication interface (130) can receive biometric data and medical data through communication with a database within a hospital environment, a cloud server performing tasks such as standardization of medical data, or a computing device. The communication interface (130) can transmit output data of a neural network model, intermediate data derived during the computation process of the processor (110), processed data, etc., via communication with the aforementioned database, server, or computing device. In particular, the processor (110) can obtain biometric data from an external biometric signal measuring device (e.g., an electrocardiogram measuring device) or a computing device (e.g., a smart watch) through the communication interface (130).

[0065] Furthermore, the communication interface (130) may transmit and receive data related to the dynamic setting of a cut-off value according to one embodiment of the present disclosure to an external device. For example, the communication interface (130) may transmit characteristic information of a medical environment to which the computing device (100) is applied to an external server, and receive from the external server anonymized operational data of other medical environments included in a cluster corresponding to the medical environment (e.g., cut-off values ​​used in other medical environments and performance indicators based thereon). Additionally, the communication interface (130) may transmit the cut-off value determined or updated by the computing device (100) and the performance indicators based on the application of the cut-off value to an external server so that they can be utilized in other medical environments.

[0066] FIG. 2 is a schematic flowchart of a control method for a computing device that dynamically sets a cut-off of a neural network model based on the characteristics of an environment according to one embodiment of the present disclosure.

[0067] Referring to FIG. 2, the processor can obtain a neural network model trained to output a score corresponding to the patient's condition based on biological data (S210).

[0068] Specifically, the processor (110) may obtain a previously trained neural network model by receiving it from an external server or external computing device through a communication interface (130), or by calling a previously stored neural network model in memory (120). Alternatively, the processor (110) may obtain a trained neural network model by training the neural network model based on a training dataset.

[0069] For example, the processor (110) inputs multiple biological data (10) (e.g., multiple electrocardiogram data) included in a training dataset as input data to a neural network model, and can calculate a loss function based on the difference between the output value for each biological data (10) output by the neural network model and the label data corresponding to each biological data (10). Here, the label data may refer to ground truth information regarding the presence or absence or severity of a specific disease of a patient corresponding to each biological data (10). The loss function may be defined as a cross-entropy loss, a binary cross-entropy loss, or a weighted loss function that considers class imbalance. The processor (110) can adjust the weights of the neural network model through backpropagation based on the calculated loss function. The processor (110) can improve the classification performance of the neural network model for a specific disease by repeating this process, and can finally obtain a neural network model trained to output a score corresponding to the presence or possibility of a patient's specific disease based on the biological data (10).

[0070] Meanwhile, neural network models can be implemented using Multi-Layer Perceptron (MLP), Convolutional Neural Networks (CNN), or Recurrent Neural Networks (RNN) structures. However, this is merely one example regarding the structure of a neural network model, and various modifications may be made within the scope understandable to those skilled in the art based on the contents of this disclosure.

[0071] In the present disclosure, the score is the output data (or output value) of a neural network model or a value obtained by applying weights to the output data, and can be interpreted as an indicator representing the probability of a specific disease. For example, the score may be expressed as a probability value between 0 and 1, a normalized value between 0 and 100, etc., but is not limited thereto.

[0072] And, the processor can determine the cut-off value of the neural network model based on the characteristics of the medical environment (S220).

[0073] Here, the characteristics of the medical environment are a concept encompassing various information for identifying the medical environment in which the neural network model operates or distinguishing it from other medical environments, and may include, for example, the size of the medical institution, the location of the medical institution, information on clinical departments, and information on medical equipment that acquired biometric data.

[0074] More specifically, the size of the medical institution can be defined by the number of beds, the average daily number of outpatients, and the type of medical institution (tertiary general hospital, general hospital, hospital, clinic, etc.), and the location of the medical institution can be defined by geographical information such as the country, region, or city where the medical institution is located. Clinical department information may be information regarding the clinical department to which the neural network model is applied (e.g., cardiology, emergency medicine, family medicine, health checkup center, etc.), and medical equipment information may be information regarding the manufacturer, model, measurement protocol, etc. of the medical equipment that acquires the biometric data (10).

[0075] Additionally, the cut-off value may be defined as a threshold value for determining the judgment result regarding the patient's condition based on the score output by the neural network model. That is, the processor (110) can determine the judgment result regarding the patient's specific disease (e.g., positive / negative, risk group / non-risk group, or severity classification, etc.) based on the result of comparing the score output by the neural network model with the cut-off value.

[0076] Even with the same neural network model, the distribution of input biometric data (10), characteristics of the patient group, and clinical requirements (e.g., relative importance of sensitivity or specificity) may differ depending on the characteristics of the medical environment to which it is applied. Therefore, the processor (110) can dynamically determine the cut-off value of the neural network model based on the characteristics of the medical environment, thereby ensuring judgment performance suitable for each medical environment while applying the same neural network model to various medical environments.

[0077] The processor (110) can determine a cutoff value based on the characteristics of the medical environment in various ways. For example, the processor (110) can identify a cluster corresponding to the medical environment based on the characteristics of the medical environment and determine a cutoff value based on operational data of a neural network model in another medical environment included in the identified cluster. Additionally, the processor (110) may determine a cutoff value based on a target performance indicator and a ROC (Receiver Operating Characteristic) curve received from the user, or it may determine a cutoff value based on a cutoff profile generated based on patient classification information. Furthermore, the processor (110) may calculate a performance indicator of the neural network model based on reading data of the medical environment accumulated after the cutoff value is applied, and update the cutoff value based on the comparison result between the calculated performance indicator and the target performance indicator. Specific details regarding this will be explained in detail with reference to the drawings described later.

[0078] The processor can determine the result of the determination of the patient's condition by analyzing the patient's biological data based on the neural network model to which the above-determined cut-off value is applied (S230).

[0079] Specifically, the processor (110) inputs the patient's biometric data (10) into a neural network model to obtain a score corresponding to the likelihood of the patient's specific disease, and can determine a judgment result regarding the patient's condition by comparing the obtained score with a determined cut-off value.

[0080] The cutoff value may be set as a single value or as multiple values. That is, there may be at least one cutoff value applied to the neural network model, and multiple cutoff values ​​may be applied if it is necessary to classify the patient's condition more finely. For example, if the patient's condition is classified into two categories, a risk group and a non-risk group, a single cutoff value may be applied. In this case, the processor (110) may determine the patient as a risk group if the score output by the neural network model is greater than or equal to the cutoff value, and determine the patient as a non-risk group if it is less than the cutoff value.

[0081] On the other hand, when classifying the patient's condition more finely according to severity, multiple cut-off values ​​may be applied. For example, when classifying the patient's condition into three risk levels of LOW, INTERMEDIATE, and HIGH, the processor (110) may apply a first cut-off value and a second cut-off value (first cut-off value < second cut-off value). In this case, the processor (110) may determine the patient's condition as LOW if the score output by the neural network model is less than the first cut-off value, determine it as INTERMEDIATE if it is greater than or equal to the first cut-off value but less than the second cut-off value, and determine it as HIGH if it is greater than or equal to the second cut-off value. When multiple cut-off values ​​are applied in this manner, the processor (110) may determine multiple cut-off values ​​together based on the characteristics of the medical environment in step S220.

[0082] The processor (110) can store the determination result of the patient's condition determined as above in memory (120), transmit it to an external device (e.g., a medical staff terminal device, a hospital information system, etc.) through a communication interface (130), or provide it to a user through an output means such as a display.

[0083] FIG. 3 is a flowchart of a control method for a computing device that identifies clusters corresponding to environmental characteristics and dynamically sets a cutoff of a neural network model according to an embodiment of the present disclosure. Since steps S310 and S340 shown in FIG. 3 correspond to steps S210 and S230 shown in FIG. 2, the above description applies equally, and a detailed description is omitted. FIG. 4 is an example diagram of a control method for a computing device that identifies clusters corresponding to environmental characteristics and dynamically sets a cutoff of a neural network model according to an embodiment of the present disclosure.

[0084] Referring to FIG. 3, the processor can identify a cluster corresponding to the medical environment based on at least one of the size of the medical institution, the location of the medical institution, clinical division information, and the medical equipment information from which the biometric data was acquired, which are included in the characteristics of the medical environment (S320).

[0085] A cluster according to one embodiment of the present disclosure may be defined as a set of medical environments having similar characteristics in determining the cutoff value of a neural network model. That is, since medical environments included in the same cluster have similar characteristics in terms of the size, location, clinical division, medical equipment, etc. of the medical institution, they can be understood as environments where similar performance indicators are highly likely to be derived even if the same or similar cutoff values ​​are applied.

[0086] A plurality of clusters according to one embodiment of the present disclosure may be pre-configured and stored in memory (120) or an external database. For example, a processor (110) may form a plurality of clusters by collecting characteristic information of each of a plurality of medical environments to which a neural network model is applied (i.e., size of the medical institution, location of the medical institution, clinical department information, medical equipment information, etc.) and by clustering the plurality of medical environments based on the collected characteristic information.

[0087] At this time, the processor (110) may form multiple clusters by utilizing various clustering algorithms such as K-means clustering, hierarchical clustering, and DBSCAN. Alternatively, according to one embodiment of the present disclosure, multiple clusters may be formed rule-based according to predefined criteria, such as segments by size of medical institutions, zones by location, groups by clinical department, and groups by medical equipment.

[0088] Referring to FIG. 4, the processor (110) can set up a plurality of clusters (421, 422, 423, 424, 425) in a pre-set coordinate space (410). Here, the coordinate space (410) may refer to a multidimensional space defined by a plurality of axes representing the characteristics of a medical environment. For example, the coordinate space (410) may be defined as a multidimensional space including an axis representing the size of the medical institution, an axis representing the location of the medical institution, an axis representing clinical divisions, an axis representing medical equipment, etc. However, for convenience of explanation in FIG. 4, the coordinate space (410) is depicted as a three-dimensional space, but the number of dimensions of the coordinate space (410) is not limited thereto.

[0089] A processor (110) according to one embodiment of the present disclosure can convert information regarding the characteristics of a medical environment into a numerical vector and identify it as a coordinate point in a coordinate space (410). Specifically, the processor (110) can convert each item included in the characteristics of the medical environment (e.g., size of the medical institution, location of the medical institution, clinical division, medical equipment, etc.) into a numerical value and convert it into a vector. For example, the processor (110) can convert categorical information (e.g., type of medical institution, clinical division, manufacturer of medical equipment, etc.) into a numerical vector by a one-hot encoding method or an embedding method, and can convert numerical or ordinal information (e.g., number of beds, daily average number of outpatients, etc.) into a numerical vector by normalizing it. The processor (110) can combine the converted numerical vectors to generate an integrated vector corresponding to the medical environment and identify the generated integrated vector as a coordinate point in a coordinate space (410).

[0090] In the coordinate space (410), each medical environment can be identified as a coordinate point corresponding to the characteristics of the medical environment, and adjacent coordinate points in the coordinate space (410) can be grouped into a single cluster. That is, a plurality of clusters (421, 422, 423, 424, 425) according to one embodiment of the present disclosure can be set to occupy different regions in the coordinate space (410). The processor (110) can identify a coordinate point (430) corresponding to the characteristics of the medical environment to which a neural network model is applied in the coordinate space (410), and can identify a cluster (e.g., cluster (423)) containing the identified coordinate point (430) as a cluster corresponding to the medical environment.

[0091] And, the processor (110) can determine a cut-off value based on operational data of a neural network model (500) in at least one other medical environment included in the cluster (S330). Here, the operational data may include a cut-off value applied to a neural network model (500) used in a different medical environment. That is, the operational data may include a cut-off value applied to a neural network model (500) by a computing device in a medical environment different from the computing device (100). In addition, the operational data may further include a cut-off value applied to a neural network model (500) in a different environment, performance indicators of the neural network model (500) calculated with the cut-off value applied (e.g., sensitivity, specificity, accuracy, positive predictive value (PPV), negative predictive value (NPV), F1 score, etc.), patient diagnosis results and clinical confirmation results accumulated in that medical environment, and patient prevalence information in that medical environment.

[0092] The processor (110) can identify a range of cutoff values ​​based on operational data of other medical environments included in the identified cluster and determine a cutoff value within the identified range of cutoff values. For example, the processor (110) can identify a range of cutoff values ​​based on the minimum value, maximum value, average value, median value, etc. of cutoff values ​​used in other medical environments included in the identified cluster. Alternatively, the processor (110) can identify a cutoff value that satisfies a pre-set performance indicator (e.g., operational data with the highest F1 score) among the operational data of other medical environments included in the identified cluster, and determine the identified cutoff value as the cutoff value of the neural network model.

[0093] Additionally, the processor (110) may output a range of identified cut-off values ​​or a user interface (UI) that allows a user to view and adjust the identified cut-off values ​​to a display device or an external terminal device connected to a communication interface (130). For example, the user interface may display a range of identified cut-off values ​​(e.g., 0.3 to 0.5) in the form of a slider and display a cut-off value most frequently applied in other medical environments included in the identified cluster (e.g., 0.4) as a recommended value. Through the user interface, the user may adopt the recommended value as is or select any value within the range of cut-off values ​​as a cut-off value by manipulating the slider. The processor (110) may determine the final cut-off value based on user input received through the user interface.

[0094] Referring again to FIG. 4, when a coordinate point (430) corresponding to the characteristics of a medical environment to which a neural network model is to be applied is identified as being included in a cluster (423), the processor (110) can acquire operational data of other medical environments (431, 432, 433) included in the cluster (423) and determine a cut-off value of the neural network model based on the acquired operational data. For example, if the cutoff value applied to the neural network model in another medical environment (431) included in the cluster (423) is 0.4 and the corresponding sensitivity is 0.85 and specificity is 0.80, if the cutoff value applied to the neural network model in another medical environment (432) is 0.45 and the corresponding sensitivity is 0.82 and specificity is 0.83, and if the cutoff value applied to the neural network model in another medical environment (433) is 0.42 and the corresponding sensitivity is 0.84 and specificity is 0.81, the processor (110) can identify a range of cutoff values ​​from 0.40 to 0.45 based on these operational data and determine the cutoff value of the neural network model within that range.

[0095] Below, the operation of the processor (110) identifying candidate clusters based on the size of the medical institution, clinical division, location of the medical institution, and medical equipment, respectively, as characteristics of the medical environment, will be described in detail.

[0096] First, the processor (110) can identify a candidate cluster (hereinafter, the first candidate cluster) based on the size of the medical institution for a plurality of different medical environments, and can identify a cluster corresponding to the medical environment based on the first candidate cluster.

[0097] Specifically, the processor (110) can identify the size of the medical institution in the medical environment to which the neural network model is to be applied (e.g., number of beds, daily average number of outpatients, type of medical institution, etc.) and identify a first candidate cluster that includes other medical environments having the same or similar size as the identified medical institution. For example, the processor (110) can identify medical environments corresponding to tertiary general hospitals as one first candidate cluster based on the type of medical institution, or identify the first candidate cluster by classifying them into large medical environments with 500 or more beds, medium medical environments with 100 or more beds and less than 500 beds, and small medical environments with less than 100 beds, respectively, based on the number of beds. Since the severity distribution of the patient group and the operating environment of the neural network model may differ depending on the size of the medical institution, identifying the first candidate cluster based on the size of the medical institution enables the use of operating data from medical environments with similar operating environments when determining the cutoff value.

[0098] Additionally, the processor (110) can identify a candidate cluster (hereinafter, a second candidate cluster) based on a clinical division, and identify a cluster corresponding to a medical environment based on the second candidate cluster.

[0099] Specifically, the processor (110) may identify a clinical division of a medical environment to which a neural network model is to be applied (e.g., cardiology, emergency medicine, family medicine, health checkup center, etc.) and identify a second candidate cluster that includes other medical environments that are identical to or similar to the identified clinical division or clinical requirements. Clinical division information according to one embodiment of the present disclosure may be classified into clinical requirement profiles according to performance characteristics required of the neural network model in the corresponding clinical division. For example, clinical requirement profiles may be classified into (i) a sensitivity-first profile in which high sensitivity is primarily required (e.g., internal medicine, oncology, emergency medicine, etc.), (ii) a specificity-first profile in which high specificity is primarily required (e.g., surgery, thoracic surgery, orthopedics, etc.), (iii) a balanced profile in which sensitivity and specificity are required in balance (e.g., cardiology, neurology, etc.), (iv) a profile of an environment with a relatively low prevalence (e.g., health checkup center, family medicine, etc.). The processor (110) can identify which clinical division of the medical environment to which the neural network model is to be applied belongs to which clinical need profile, and can identify other medical environments belonging to the same clinical need profile as second candidate clusters.

[0100] Additionally, the processor (110) can identify a candidate cluster (hereinafter, a third candidate cluster) based on the location of the medical institution, and identify a cluster corresponding to the medical environment based on the third candidate cluster.

[0101] Specifically, the processor (110) can identify the location of a medical institution in a medical environment to which a neural network model is to be applied (e.g., country, region, city, etc.) and identify a third candidate cluster that includes other medical environments that are identical to or geographically adjacent to the location of the identified medical institution. Since the racial and genetic characteristics of the patient group, regional prevalence, and medical practices may differ depending on the location of the medical institution, identifying a third candidate cluster based on the location of the medical institution enables the use of operational data from medical environments with similar patient group characteristics when determining a cutoff value. For example, the processor (110) can identify medical environments belonging to the same country as a third candidate cluster, or medical environments belonging to the same region (e.g., East Asia, North America, Western Europe, etc.) as a third candidate cluster. Alternatively, the processor (110) can classify the location of the medical institution into urban, urban, and rural regional divisions and identify other medical environments belonging to the same regional division as a third candidate cluster.

[0102] Additionally, the processor (110) can identify a candidate cluster (hereinafter, a fourth candidate cluster) based on the medical equipment that acquired the biometric data (10), and can identify a cluster corresponding to a medical environment based on the fourth candidate cluster.

[0103] Specifically, information regarding medical equipment that has acquired biodata (10) according to one embodiment of the present disclosure may include at least one of the manufacturer of the medical equipment, the model of the medical equipment, and a measurement protocol (or imaging protocol) set on the medical equipment. The manufacturer and model of the medical equipment may include, for example, manufacturers and models of imaging equipment such as CT, MRI, PET-CT (e.g., GE, Siemens, Canon, etc.), manufacturers and models of endoscope equipment (e.g., Olympus, Fujifilm, Pentax, etc.), manufacturers and models of electrocardiogram measurement equipment, etc. Additionally, the measurement protocol or imaging protocol set on the medical equipment may include, for example, in the case of imaging equipment, settings for imaging conditions such as whether to use a contrast agent, slice thickness, and shooting angle; in the case of endoscope equipment, settings for resolution and color profile; and in the case of electrocardiogram measurement equipment, settings for sampling frequency, filtering, lead configuration, etc.

[0104] When the manufacturer, model, measurement protocol, etc. of the medical equipment are different, even if the biometric data (10) is measured from the same patient, the signal characteristics, noise level, sampling frequency, image resolution, and color profile may differ, and accordingly, the distribution of scores output by the neural network model may also differ. For example, even if the biometric data (10) is measured from the same lesion of the same patient, the score output by the neural network model for the biometric data (10) obtained from the medical equipment of the first manufacturer may be 0.35, whereas the score output by the neural network model for the biometric data (10) obtained from the medical equipment of the second manufacturer may be 0.42.

[0105] In addition, even if medical equipment of the same model from the same manufacturer is used, if the measurement protocol is different (for example, if the use of contrast agent is different in the same CT equipment, or if the sampling frequency is different in the same electrocardiogram measuring equipment, etc.), the characteristics of the acquired bio-data (10) may differ, and the score distribution of the neural network model may appear differently.

[0106] Accordingly, the processor (110) can identify a fourth candidate cluster that includes other similar medical environments where the similarity of the distribution of scores output from the neural network model is greater than or equal to a preset value, based on information about medical equipment. For example, the processor (110) can calculate a similarity (e.g., KL divergence (Kullback-Leibler divergence), cosine similarity, distance between distributions, etc.) by comparing the score distribution of the neural network model for biometric data (10) obtained from medical equipment in a medical environment to which the neural network model is applied with the score distribution of the neural network model for biometric data (10) obtained from medical equipment in a different medical environment. If the calculated similarity is greater than or equal to a preset value, the processor (110) can include the other medical environment in the fourth candidate cluster.

[0107] Meanwhile, the processor (110) may identify the type of biometric data (10) obtained in a medical environment to which a neural network model is applied, and may identify a fourth candidate cluster based on information about medical equipment corresponding to the identified type of biometric data (10). Here, the type of biometric data (10) refers to the type of biometric data (10) obtained through medical equipment, and may include, for example, electrocardiogram data, electroencephalogram data, photoplethysmography data, MRI images, CT images, endoscopic images, etc. Since the type of medical equipment used to obtain the biometric data (10) and the items of the measurement protocol set on the medical equipment may differ depending on the type of biometric data (10), the processor (110) may identify information about medical equipment differentially according to the identified type of biometric data (10).

[0108] For example, if the type of identified biometric data (10) is electrocardiogram data, the processor (110) can identify the manufacturer and model of the electrocardiogram measuring device and the measurement protocol set in the electrocardiogram measuring device (e.g., sampling frequency, filtering settings, lead configuration, etc.) as information about the medical device. Additionally, if the type of identified biometric data (10) is a CT image, the processor (110) can identify the manufacturer and model of the CT device (e.g., GE, Siemens, Canon, etc.) and the imaging protocol set in the CT device (e.g., whether a contrast agent is used, slice thickness, imaging angle, tube voltage, tube current, etc.) as information about the medical device. Additionally, if the type of identified biological data (10) is an endoscopic image, the processor (110) can identify the manufacturer and model of the endoscopic equipment (e.g., Olympus, Fujifilm, Pentax, etc.) and the shooting protocol set on the endoscopic equipment (e.g., resolution, color profile, light source settings, etc.) as information about the medical equipment. Additionally, if the type of identified biological data (10) is an MRI image, the processor (110) can identify the manufacturer and model of the MRI equipment and the shooting protocol set on the MRI equipment (e.g., magnetic field strength, pulse sequence, slice thickness, etc.) as information about the medical equipment.

[0109] Based on information regarding medical equipment that is differentially identified according to the type of biometric data (10) as described above, the processor (110) can identify other medical environments that are similar to the medical environment to which the neural network model is applied and the medical equipment environment as a fourth candidate cluster.

[0110] Meanwhile, the processor (110) may identify a cluster corresponding to the medical environment by sequentially performing identification operations of candidate clusters based on the size of the medical institution, clinical division, location of the medical institution, and medical equipment, respectively.

[0111] Specifically, the processor (110) can identify a first candidate cluster based on the size of the medical institution for a plurality of different medical environments, identify a second candidate cluster based on the clinical division for the identified first candidate cluster, identify a third candidate cluster based on the location of the medical institution for the identified second candidate cluster, and identify a fourth candidate cluster based on the medical equipment that acquired the biometric data (10) for the identified third candidate cluster, thereby determining the identified fourth candidate cluster as a cluster corresponding to the medical environment. That is, the processor (110) can identify a cluster composed of other medical environments that are most similar to the medical environment to which the neural network model is applied by first narrowing the candidate group based on the size of the medical institution for a population of a plurality of different medical environments, secondarily narrowing the candidate group based on the clinical division for the narrowed candidate group, thirdly narrowing the candidate group based on the location of the medical institution for the narrowed candidate group, and finally narrowing the candidate group based on the medical equipment.

[0112] This sequential identification method may correspond to an identification method where the priority of medical environment characteristics for identifying clusters is set in the order of the size of the medical institution, clinical divisions, location of the medical institution, and medical equipment.

[0113] Additionally, the processor (110) may identify a cluster corresponding to a medical environment by first performing an identification operation of a candidate cluster based on the medical equipment that acquired the biometric data (10).

[0114] Specifically, the processor (110) can determine the identified third candidate cluster as a cluster corresponding to a medical environment by first identifying a fourth candidate cluster based on medical equipment that has acquired biometric data (10) for a plurality of different medical environments, identifying a first candidate cluster based on the size of the medical institution for the identified fourth candidate cluster, identifying a second candidate cluster based on the clinical division for the identified first candidate cluster, and identifying a third candidate cluster based on the location of the medical institution for the identified second candidate cluster.

[0115] Since differences in the manufacturer, model, or measurement protocol of medical equipment can directly affect the score distribution output by the neural network model, prioritizing the identification of candidate clusters based on medical equipment allows for the preferential grouping of other medical environments with similar neural network model score distributions, followed by the gradual narrowing of the candidate pool based on remaining characteristics. Consequently, it becomes possible to determine cutoff values ​​based on operational data from medical environments with similar neural network model output characteristics, thereby improving the accuracy of cutoff value determination.

[0116] However, the order of identifying candidate clusters is not limited to the two examples above, and the identification of candidate clusters based on the size of the medical institution, clinical department, location of the medical institution, and medical equipment can be performed in any combination order. For example, if the neural network model is a model trained to output a score corresponding to the possibility of Left Ventricular Systolic Dysfunction (LVSD) from a patient's electrocardiogram data, the processor (110) can identify clusters corresponding to the medical environment based on the size of the medical institution in the medical environment (e.g., a tertiary general hospital), the location of the medical institution (e.g., Seoul), clinical department (e.g., cardiology), medical equipment (e.g., 12-lead electrocardiogram measuring equipment), etc. If the cutoff value applied to the neural network model for the determination of LVSD in other medical environments included in the identified cluster and the corresponding performance indicators are accumulated as operational data, the processor (110) can determine the cutoff value for the determination of LVSD in the neural network model of its own medical environment based on the operational data.

[0117] Meanwhile, the processor (110) may perform an adaptive identification operation in the sequential identification process of the candidate clusters described above, if the number of medical environments included in the candidate clusters of each stage decreases below a preset threshold value, by relaxing the identification conditions of that stage or omitting the identification operation of that stage. Here, the adaptive identification operation may mean an operation that allows the identification operation of a subsequent stage to be performed normally by changing the identification conditions of that stage or omitting the identification operation of that stage itself when the number of medical environments included in the candidate clusters falls below the minimum level required for determining the cut-off value. Meanwhile, the adaptive identification operation of the present disclosure may be referred to, for example, as a fallback operation.

[0118] Specifically, the processor (110) can compare the number of other medical environments included in the candidate cluster identified at each stage with a preset threshold (e.g., 5). If the number of other medical environments included in the candidate cluster is greater than or equal to the threshold, the processor (110) can perform the candidate cluster identification operation of the next stage. On the other hand, if the number of other medical environments included in the candidate cluster is less than the threshold, the processor (110) can relax the identification conditions of that stage to re-identify the candidate cluster, or skip the identification operation of that stage and perform the candidate cluster identification operation of the next stage.

[0119] For example, if the processor (110) identifies a third candidate cluster based on the location of a medical institution and the number of other medical environments included in the third candidate cluster is less than a threshold value, it may re-identify the third candidate cluster by relaxing the identification condition for the location of the medical institution (e.g., expanding the identification condition from "same city" to "same region" or "same country"), or it may omit the candidate cluster identification operation based on the location of the medical institution and perform the next step of candidate cluster identification operation (e.g., identification of a fourth candidate cluster based on medical equipment) for the candidate cluster identified in the previous step.

[0120] According to the adaptive identification operation described above, even if there are not enough other medical environments that exactly match all characteristics with the medical environment to which the neural network model is applied, the processor (110) can identify valid clusters corresponding to the medical environment and maintain the reliability of the cutoff value determination.

[0121] FIG. 5 is a flowchart of a method for updating a cutoff value based on a performance indicator of a neural network model according to a determined cutoff value according to an embodiment of the present disclosure. Since steps S510, S520, and S530 shown in FIG. 5 correspond to steps S210, S220, and S230 shown in FIG. 2, the above description applies equally, and a detailed description is omitted. FIG. 6 is an example diagram of a method for updating a cutoff value based on a performance indicator of a neural network model according to a determined cutoff value according to an embodiment of the present disclosure.

[0122] Meanwhile, the cutoff value applied to the neural network model (500) according to one embodiment of the present disclosure may be determined based on operational data of another medical environment included in a cluster corresponding to a medical environment as described in FIGS. 3 and 4, or may be applied as a default cutoff value pre-set in the neural network model (500). Here, the default cutoff value refers to a basic cutoff value pre-set at the time of training of the neural network model (500), for example, it may be set as a cutoff value in which the performance indicator for the training dataset in which the neural network model (500) is trained is optimized.

[0123] In this way, when a pre-set default cut-off value is applied to the neural network model (500), the cut-off value needs to be corrected to a value suitable for the medical environment based on the reading data accumulated in the medical environment where the neural network model (500) is applied.

[0124] Below, the operation of updating the cut-off value based on accumulated reading data after the determined cut-off value (e.g., a cluster-based cut-off value or a default cut-off value) is applied to the neural network model (500) is described in detail.

[0125] Referring to FIG. 5, the processor (110) can calculate the performance indicator of the neural network model (500) based on reading data of the medical environment accumulated over a preset period after the determined cut-off value is applied to the neural network model (500) (S540).

[0126] Here, the reading data in the medical environment may include a patient's diagnosis result based on a score output from a neural network model (500) and a clinical confirmation result corresponding to the diagnosis result. That is, the reading data may consist of a pair of an artificial intelligence diagnosis result based on the neural network model (500) and a confirmation result confirmed in actual clinical practice for the patient. The clinical confirmation result may refer to actual positive / negative or severity information confirmed based on, for example, additional test results by medical staff, tissue biopsy results, final reading results by an imaging specialist, patient follow-up results, etc.

[0127] The pre-set period may be set to a period during which a sufficient amount of reading data can be accumulated to determine the appropriateness of the cutoff value. For example, the pre-set period may be set to a unit period such as 1 month, 3 months, or 6 months, or it may be set to a period until the number of accumulated reading data reaches a pre-set number (e.g., 100 cases, 500 cases, etc.). However, the length of the pre-set period is not limited to this and may be set in various ways considering the characteristics of the medical environment, the volume of patients, and the frequency of examinations by clinical department.

[0128] The processor (110) can calculate the number of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN) by comparing the judgment result of the neural network model (500) with the clinical confirmation result for each pair of reading data accumulated over a preset period. The processor (110) can calculate the performance indicator of the neural network model (500) based on the calculated number of True Positives, True Negatives, False Positives, and False Negatives.

[0129] And, the processor (110) can update the cut-off value by comparing the calculated performance indicator with the preset target performance indicator (S550).

[0130] Here, the pre-set target performance indicator may refer to a performance level required in a medical environment where the neural network model (500) is operated, and may be pre-set by a user (e.g., an administrator of a medical institution, medical staff, etc.) or may be automatically set based on performance indicators in other medical environments included in a cluster corresponding to the medical environment. For example, the target performance indicator may be set as a target value for one or more performance indicators, such as sensitivity 0.85 or higher, specificity 0.80 or higher, F1 score 0.82 or higher, etc.

[0131] In one embodiment of the present disclosure, the cutoff value applied to the neural network model (500) is, for example, a processor (110) inputs electrocardiogram data into the neural network model (500) and compares the score output from the neural network model (500) with a threshold value for identifying a high-risk group (HIGH) regarding the patient's left ventricular systolic dysfunction (LVSD) to determine the user's condition. That is, if the score output by the neural network model (500) is greater than or equal to the cutoff value, the patient may be determined to be in the high-risk group for left ventricular systolic dysfunction, and if it is less than the cutoff value, the patient may be determined not to be in the high-risk group. However, the cutoff value is not limited to identifying the high-risk group for left ventricular systolic dysfunction and may also be applied to identifying other risk levels of a specific disease (e.g., LOW, INTERMEDIATE, etc.) or determining other specific diseases.

[0132] The processor (110) can maintain the currently applied cut-off value if the calculated performance indicator meets the target performance indicator. On the other hand, if the calculated performance indicator falls short of the target performance indicator, the processor (110) can update the cut-off value in various ways.

[0133] First, if the calculated performance indicator falls short of the target performance indicator, the processor (110) can repeatedly calculate the performance indicator while adjusting the cutoff value in steps by a preset unit, and update it to the cutoff value that first satisfies the target performance indicator.

[0134] Specifically, referring to FIG. 6, the processor (110) can adjust the cutoff value upward or downward by a preset unit (e.g., 0.01, 0.05, etc.) based on the currently applied cutoff value. The direction of adjustment of the cutoff value can be determined according to the type of performance indicator that is not met. For example, if sensitivity is not met at the target performance indicator, the processor (110) can adjust the cutoff value downward so that the neural network model (500) determines positive more aggressively, and if specificity is not met at the target performance indicator, the processor (110) can adjust the cutoff value upward so that the neural network model (500) determines positive more conservatively.

[0135] For example, assume that a neural network model (500) is a model that outputs a score corresponding to left ventricular systolic dysfunction from a patient's electrocardiogram data, and that the cutoff value currently applied to the neural network model (500) is set to 85 as a threshold for identifying a high-risk group for left ventricular systolic dysfunction. In this case, as a result of operating the neural network model (500) in a medical environment, the sensitivity to accumulated reading data is calculated to be 0.75, and false negative cases may frequently occur in which patients belonging to the high-risk group for left ventricular systolic dysfunction are judged negative. In such cases, the processor (110) can derive a cutoff value in which the sensitivity meets the target performance indicator by gradually lowering the cutoff value to reduce false negatives in patients with left ventricular systolic dysfunction.

[0136] In this regard, referring to FIG. 6, if the currently applied cut-off value (610) is 85 and the corresponding sensitivity is 0.75, which is less than the target performance indicator of 0.85, the processor (110) can lower the cut-off value by a preset unit (e.g., 0.05) to change it to 84.95, and virtually apply the changed cut-off value (620) to the neural network model (500) to recalculate the performance indicator for the accumulated reading data. If the recalculated performance indicator still falls short of the target performance indicator (e.g., if the sensitivity is calculated as 0.81), the processor (110) can lower the cut-off value again by 0.05 to change it to 84.90, and recalculate the performance indicator for the changed cut-off value (630). As a result of repeating this process, when the cutoff value is 84.90 and the sensitivity is calculated as 0.86, it is confirmed that the target performance indicator is satisfied for the first time, the processor (110) can update the cutoff value of the neural network model (500) to 0.30.

[0137] Additionally, the processor (110) may stop adjusting the cutoff value if the adjusted cutoff value falls outside a preset allowable range during the process of adjusting the cutoff value stepwise. Here, the allowable range refers to a range defined by the minimum and maximum values ​​that the cutoff value of the neural network model (500) can have, for example, a range of 0.1 to 0.9. If the adjusted cutoff value falls outside the allowable range, the processor (110) determines that the target performance indicator cannot be satisfied by adjusting the cutoff value alone and may output a notification to the user recommending retraining of the neural network model (500) or resetting of the target performance indicator.

[0138] Meanwhile, the processor (110) can monitor other performance indicators in addition to the performance indicator being adjusted during the process of gradually adjusting the cutoff value. That is, when adjusting the cutoff value of the neural network model (500), a trade-off relationship may occur in which the performance indicator being adjusted is improved, but other performance indicators are degraded. For example, if the cutoff value is adjusted upward, specificity may increase but sensitivity may decrease, and if the cutoff value is adjusted downward, sensitivity may increase but specificity may decrease.

[0139] Accordingly, the processor (110) can determine whether the performance indicator to be adjusted meets the target performance indicator while adjusting the cutoff value stepwise, and whether other performance indicators not to be adjusted are maintained above a preset allowable lower limit. For example, while lowering the cutoff value to bring sensitivity to the target performance indicator (e.g., 0.85 or higher), the processor (110) can determine whether the specificity is maintained above a preset allowable lower limit (e.g., 0.75 or higher). If the specificity drops below the allowable lower limit even though the sensitivity meets the target performance indicator for the stepwise adjusted cutoff value, the processor (110) may withhold updating to that cutoff value and output a notification recommending to the user to reduce the adjustment range of the cutoff value or readjust the target performance indicator itself. According to this operation, the processor (110) can update the cutoff value while maintaining a balance among multiple performance indicators.

[0140] Additionally, the processor (110) may generate and provide to the user a summary report that comprehensively organizes information related to the update of the cut-off value. Here, the summary report may include the currently applied cut-off value, performance indicators based on the currently applied cut-off value, recommended update cut-off value, expected performance indicators based on the recommended update cut-off value (e.g., expected sensitivity, expected specificity, expected accuracy, expected PPV, expected NPV, etc.), and the reason for updating the cut-off value. The processor (110) may output the generated summary report to a display device or an external terminal device connected to the communication interface (130).

[0141] At this time, the processor (110) can update the cutoff value of the neural network model (500) to a recommended update cutoff value only when user approval input for the summary report is received. That is, the processor (110) does not automatically update the cutoff value, but updates the cutoff value through approval from a user (e.g., medical staff, administrator of a medical institution, etc.). According to this operation, considering that the update of the cutoff value is a matter affecting medical practice in the clinical field, clinical accountability regarding the update of the cutoff value can be ensured, and the cutoff value can be updated under the control of the user.

[0142] Meanwhile, if the calculated performance indicator falls short of the target performance indicator, the processor (110) may update the cutoff value by re-determining the cutoff value based on operational data from other medical environments included in the cluster corresponding to the medical environment. Specifically, the processor (110) may re-obtain the cutoff value applied to the neural network model (500) and the corresponding performance indicator from other medical environments included in the cluster corresponding to the medical environment, and re-determine the cutoff value of the neural network model (500) based on the obtained operational data. That is, even after the cutoff value is determined in the medical environment, operational data from other medical environments included in the cluster can be accumulated and updated, so the processor (110) can re-determine the cutoff value based on the latest operational data. According to this update method, the processor (110) can update the cutoff value by reflecting not only the medical environment where the neural network model (500) is applied but also the operational experience from other medical environments belonging to the same cluster.

[0143] Meanwhile, the processor (110) may store update history information in memory (120) whenever the cutoff value is updated. Here, the update history information may include the cutoff value prior to the update, the cutoff value after the update, the time of the update of the cutoff value, the performance indicator calculated at the time of the update, the reason for the update (e.g., automatic update based on stepwise adjustment, re-determination based on cluster operation data, update by user input, etc.). Based on the update history information stored in memory (120), the processor (110) may track the trend of change of the cutoff value, analyze the performance change of the neural network model (500) due to the update of the cutoff value, or output the update history information through a user interface according to a user's request.

[0144] According to the above-described update operation of the cut-off value, the processor (110) can continuously optimize the cut-off value based on actual reading data accumulated in a medical environment where the neural network model (500) is applied, and accordingly, the neural network model (500) can be adaptedly operated in the medical environment.

[0145] FIG. 7 is a flowchart of a method for determining a cut-off value based on an ROC curve according to an embodiment of the present disclosure. S710 and S750 shown in FIG. 7 correspond to S210 and S230 shown in FIG. 2, and the above description applies equally, and a detailed description is omitted.

[0146] According to one embodiment of the present disclosure, the processor (110) receives a target performance indicator for a neural network model (500) from a user (S720), obtains a Receiver Operating Characteristic (ROC) curve for the neural network model (500) (S730), and can determine a cut-off value that satisfies the input target performance indicator based on the ROC curve (S740).

[0147] Specifically, the processor (110) can obtain target performance indicators for the neural network model (500) from the user through a communication interface (130), a display, or a user interface. For example, the user can input target values ​​for performance indicators that the neural network model (500) must achieve through a user interface displayed on a display. For example, the user can input target values ​​for one or more performance indicators through the user interface, such as sensitivity 0.85 or higher, specificity 0.80 or higher, accuracy 0.85 or higher, F1 score 0.82 or higher, etc. The processor (110) can receive the target performance indicators input through the user interface.

[0148] Additionally, the processor (110) may obtain an ROC curve for the neural network model (500). Here, the ROC curve may be defined as a curve plotted in coordinate space between pairs of sensitivity and specificity (i.e., 1 minus specificity) calculated while varying the cutoff value applicable to the neural network model (500). Each point on the ROC curve corresponds to a specific cutoff value, and each point on the ROC curve may represent the sensitivity and specificity achieved by the neural network model (500) when the corresponding cutoff value is applied.

[0149] According to one embodiment of the present disclosure, an ROC curve can be calculated based on pairs of scores output by a neural network model (500) and correct labels corresponding to the scores (e.g., clinical confirmation results, etc.). A processor (110) can calculate an ROC curve by calculating sensitivity and specificity for each cut-off value while gradually changing the cut-off value for the pairs of scores and correct labels of the neural network model (500) within a range from 0 to 1.

[0150] A processor (110) according to one embodiment of the present disclosure may obtain a pre-calculated ROC curve for a training dataset or a verification dataset at the time of training of a neural network model (500). Alternatively, the processor (110) may obtain a pre-calculated ROC curve by receiving it from an external server or an external computing device through a communication interface (130).

[0151] The processor (110) can determine a cutoff value that satisfies the input target performance indicator based on the acquired ROC curve. Specifically, the processor (110) can determine whether the sensitivity and specificity corresponding to each point on the ROC curve satisfy the input target performance indicator. The processor (110) can identify a point among the points on the ROC curve that satisfies the input target performance indicator and determine the cutoff value corresponding to the identified point as the cutoff value of the neural network model (500).

[0152] For example, if the neural network model (500) is a model that outputs a score corresponding to left ventricular systolic dysfunction from a patient's electrocardiogram data, and the user inputs a sensitivity of 0.90 or higher as a target performance indicator so as not to miss patients with left ventricular systolic dysfunction, the processor (110) can identify a point on the ROC curve for the neural network model (500) that satisfies a sensitivity of 0.90 or higher, and determine a cutoff value corresponding to the identified point as a cutoff value for determining left ventricular systolic dysfunction.

[0153] As another example, if the user inputs a target performance indicator of 0.85 or higher and a specificity of 0.80 or higher, the processor (110) can identify points on the ROC curve that have a sensitivity of 0.85 or higher and a specificity of 0.20 or lower. If there are multiple identified points, the processor (110) can determine a cutoff value corresponding to a point satisfying a preset condition among the multiple identified points as the cutoff value of the neural network model (500). For example, the processor (110) can determine a cutoff value corresponding to the point closest to the upper left of the ROC curve (i.e., a point where both sensitivity and specificity are close to 1) among the multiple identified points as the cutoff value of the neural network model (500), or determine a cutoff value corresponding to the point with the highest F1 score as the cutoff value of the neural network model (500).

[0154] Alternatively, the processor (110) may output a user interface to a display device or external terminal device connected to a communication interface (130), the user interface including a cutoff value corresponding to each of the identified multiple points and an expected performance indicator (e.g., expected sensitivity, expected specificity, expected accuracy, expected PPV, expected NPV, etc.) according to each cutoff value. Through the user interface, the user can compare and review the expected performance indicators according to the identified multiple cutoff values ​​and select a cutoff value suitable for the clinical situation of the medical environment. The processor (110) may determine the cutoff value of the neural network model (500) based on user input received through the user interface. According to this operation, the user can finally select a cutoff value that fits the clinical priority of the medical environment while directly reviewing the tradeoff between sensitivity and specificity.

[0155] Alternatively, if there is no point among the points on the ROC curve that satisfies the input target performance indicator, the processor (110) may output a notification to the user that the target performance indicator exceeds the performance range achievable by the neural network model (500) and recommend that the user re-enter the target performance indicator.

[0156] At this time, according to one embodiment of the present disclosure, the processor (110) may obtain an ROC curve corresponding to the cluster based on operational data of a neural network model (500) in at least one other medical environment included in the identified cluster as described in FIGS. 3 and 4.

[0157] Specifically, as described in FIGS. 3 and 4, when the processor (110) identifies a cluster corresponding to a medical environment based on the characteristics of the medical environment, the processor (110) may acquire operational data including pairs of scores output by a neural network model (500) and clinical confirmation results corresponding to those scores in each of the other medical environments included in the identified cluster. The processor (110) may calculate an ROC curve corresponding to the cluster based on the pairs of scores and clinical confirmation results included in the acquired operational data.

[0158] Alternatively, the processor (110) may obtain a pre-calculated ROC curve for each of the other medical environments included in the identified cluster, and calculate an ROC curve corresponding to the cluster by averaging or synthesizing the multiple obtained ROC curves.

[0159] In this way, when an ROC curve corresponding to a cluster is obtained, the processor (110) can determine a cutoff value that satisfies the input target performance indicator based on the ROC curve corresponding to the cluster. That is, the processor (110) can determine a cutoff value that more accurately reflects the actual performance characteristics of the medical environment to which the neural network model (500) is to be applied by utilizing an ROC curve based on operational data in a different medical environment similar to the medical environment to which the neural network model (500) is to be applied, rather than an ROC curve for the training dataset to which the neural network model (500) was trained.

[0160] Meanwhile, according to one embodiment of the present disclosure, the processor (110) may update the cut-off value applied to the neural network model (500) based on the ROC curve. Specifically, after the cut-off value is applied to the neural network model (500) as described in FIGS. 5 and 6, the processor (110) may recalculate the ROC curve based on reading data accumulated in a medical environment (i.e., pairs of scores output by the neural network model (500) and clinical confirmation results corresponding to those scores). That is, since the ROC curve calculated at the time of training of the neural network model (500) or the ROC curve corresponding to the cluster may not accurately reflect the performance actually achieved in the medical environment where the neural network model (500) is applied, the processor (110) may obtain an ROC curve that reflects the actual performance characteristics of the neural network model (500) in the medical environment by recalculating the ROC curve based on the actual reading data accumulated in the medical environment.

[0161] The processor (110) can re-identify points among the points on the recalculated ROC curve that satisfy the input target performance indicators and update the cutoff value of the neural network model (500) with a cutoff value corresponding to the identified points. According to this update operation, the processor (110) can re-evaluate the performance characteristics of the neural network model (500) based on actual reading data accumulated in a medical environment and update the cutoff value by reflecting the re-evaluated performance characteristics, so that the neural network model (500) can be operated more appropriately in the medical environment.

[0162] Additionally, the processor (110) can recalculate the ROC curve and determine whether to update the cutoff value whenever the amount of reading data accumulated in the medical environment exceeds a preset threshold (e.g., 100 cases, 500 cases, etc.) or whenever a preset period (e.g., 1 month, 3 months, etc.) arrives. According to this operation, the processor (110) can dynamically update the cutoff value while continuously tracking changes in the performance of the neural network model (500) in the medical environment.

[0163] FIG. 8 is an example diagram of a User Interface (UI) displaying distribution information of cut-off values ​​and performance indicators used in a plurality of different medical environments according to one embodiment of the present disclosure.

[0164] According to one embodiment of the present disclosure, the processor (110) may obtain anonymized operational data from a plurality of different medical environments. Here, the anonymized operational data may refer to operational data from which information capable of identifying each of the plurality of different medical environments (e.g., the name of the medical institution, the identification number of the medical institution, the identification information of the medical staff, etc.) has been removed. That is, the anonymized operational data may include cut-off values ​​and corresponding performance indicators applied to the neural network model (500) in each of the plurality of different medical environments, but may be processed data such that it is not possible to identify which medical environment the operational data originated from. By utilizing such anonymized operational data, the processor (110) can protect sensitive information of each medical environment (e.g., patient information, the operational status of the medical institution, etc.) while referencing the operational experience of the plurality of different medical environments.

[0165] And, the processor (110) can calculate distribution information of cut-off values ​​and performance indicators used in multiple different medical environments based on acquired operational data. Here, the distribution information may refer to statistical information indicating how the cut-off values ​​applied to the neural network model (500) in multiple different medical environments are distributed in a certain range and frequency, and how the performance indicators (e.g., sensitivity, specificity, accuracy, PPV, NPV, F1 score, etc.) according to each cut-off value are distributed in a certain range and frequency.

[0166] For example, distribution information may include the mean, median, standard deviation, minimum, maximum, quartiles, and frequency distribution of cutoff values, and may also include distribution information in the form of a scatter plot showing the correlation between cutoff values ​​and performance indicators.

[0167] Additionally, the processor (110) can output a user interface (UI) that displays the cut-off value determined in the medical environment where the neural network model (500) is to be applied, along with the calculated distribution information, and the performance indicators of the neural network model (500), so that they can be identified. At this time, the user interface may be output through a display included in the computing device (100) or a display device connected to the computer device (100).

[0168] Referring to FIG. 8, a user interface (800) according to one embodiment of the present disclosure may include a first area (810) that visually displays the distribution of cutoff values ​​used in a plurality of different medical environments, and a second area (820) that visually displays the distribution of performance indicators in a plurality of different medical environments. For example, the distribution of cutoff values ​​used in a plurality of different medical environments may be displayed in the first area (810) in the form of a histogram, a box plot, or a scatter plot, and the distribution of performance indicators (e.g., sensitivity, specificity, etc.) in a plurality of different medical environments may be displayed in a similar visualization form in the second area (820). Alternatively, the user interface (800) may display the distribution information in the form of an integrated chart (830) in which operational data of each of a plurality of different medical environments is plotted as data points on a coordinate space in which the cutoff values ​​are on the horizontal axis and the performance indicators are on the vertical axis.

[0169] Distribution information according to one embodiment of the present disclosure may include a quantile-based benchmarking indicator that analyzes performance indicators of a plurality of different medical environments in quantile units. Here, a quantile refers to a group formed by sorting a plurality of different medical environments according to the ranking of performance indicators and then dividing them into a certain proportion. For example, it may refer to each of the groups formed by sorting a plurality of different medical environments in order of sensitivity and then dividing them into a group of medical environments corresponding to the top 25%, a group of medical environments corresponding to the top 25-50%, a group of medical environments corresponding to the top 50-75%, a group of medical environments corresponding to the top 75-100%, etc.

[0170] Quantum-based benchmarking indicators may refer to indicators that statistically aggregate operational data of medical environments belonging to each quantile classified as above. For example, a processor (110) may sort multiple different medical environments in order of sensitivity, identify a group of medical environments corresponding to the top N% (e.g., top 20%), and generate quantum-based benchmarking indicators by calculating the average sensitivity, average specificity, average cutoff value, etc. of the medical environments belonging to the identified group.

[0171] In this way, when utilizing quantile-based benchmarking indicators, the user can determine which quantile among multiple different medical environments the current performance of the medical environment to which the neural network model (500) is applied corresponds to, and can adjust the cutoff value of their own medical environment by referring to the cutoff values ​​and performance indicators of medical environments belonging to the upper quantile. For example, if the current sensitivity of the medical environment to which the neural network model (500) is applied corresponds to the lower quantile among multiple different medical environments, the user can improve the performance of the medical environment by adjusting the cutoff value of their own medical environment to be close to the average cutoff value of medical environments belonging to the upper quantile.

[0172] That is, the user interface (800) according to one embodiment of the present disclosure may have the nature of a dynamic benchmarking report calculated by aggregating the operational experience of a plurality of different medical environments. According to this operation, the user does not merely evaluate the cutoff value determined in their own medical environment, but can dynamically refer to the operational experience of other medical environments belonging to the same cluster to adjust the cutoff value of their own medical environment to conform to standards in the medical field.

[0173] Additionally, the user interface (800) can display the cut-off value determined in the medical environment where the neural network model (500) is to be applied and the performance indicator of the neural network model (500) together with distribution information so that they can be identified. For example, the user interface (800) can display the location corresponding to the cut-off value determined in the medical environment through a separate identification marker (840) (e.g., an accent color, a separate shape, an arrow, a dotted line, etc.) on the area where the distribution information is displayed. Through this, the user can visually confirm which location the cut-off value determined in the medical environment where the neural network model (500) is to be applied corresponds to among the distribution of cut-off values ​​used in multiple other medical environments, and can visually confirm which location the performance indicator of the neural network model (500) in the medical environment corresponds to among the distribution of performance indicators in other medical environments.

[0174] Additionally, the user interface (800) may include an input means (850) (e.g., a slider, an input field, a button, etc.) capable of updating the cut-off value. The user may compare and review the relative position of the current cut-off value of the medical environment with the distribution information displayed on the user interface (800) and input a new cut-off value through the input means (850). For example, if the current cut-off value of the medical environment deviates significantly from the center of the distribution of cut-off values ​​used in multiple other medical environments, the user may adjust the cut-off value closer to the center of the distribution through the input means (850).

[0175] Additionally, the processor (110) can update the cutoff value determined in the medical environment based on user input received through the user interface (800). That is, the processor (110) can update the cutoff value of the neural network model (500) with the new cutoff value entered by the user through the input means (850).

[0176] For example, if the neural network model (500) is a model that outputs a score corresponding to left ventricular systolic dysfunction from a patient's electrocardiogram data, the user interface (800) displays the distribution of cutoff values ​​applied to the neural network model (500) for determining left ventricular systolic dysfunction in each of the other medical environments belonging to the same cluster, and the corresponding sensitivity and specificity distributions, and can visually display through an identification marker (840) where the cutoff value for determining left ventricular systolic dysfunction determined in one's own medical environment and the corresponding performance indicator correspond to the distribution of the other medical environments. Through this, the user can intuitively check the relative level of the left ventricular systolic dysfunction determination performance in one's own medical environment compared to other medical environments within the same cluster.

[0177] At this time, according to one embodiment of the present disclosure, a plurality of other medical environments may be medical environments included in the cluster identified in FIGS. 3 and FIGS. 4. That is, the processor (110) may acquire anonymized operational data of other medical environments included in the cluster identified based on the characteristics of the medical environment as described in FIGS. 3 and FIGS. 4, and may calculate distribution information based on the acquired operational data. Accordingly, the distribution information displayed on the user interface (800) may be distribution information that reflects the operational experience of other medical environments having characteristics similar to the medical environment to which the neural network model (500) is applied. According to this operation, the user may refer to the distribution of cutoff values ​​and performance indicators in other medical environments having characteristics similar to the medical environment to which the neural network model (500) is applied, and more precisely determine or update the cutoff value suitable for the medical environment.

[0178] FIG. 9 is a flowchart of a method for determining a cut-off value based on a cut-off profile corresponding to a patient according to an embodiment of the present disclosure. Since steps S910 and S950 shown in FIG. 9 correspond to steps S210 and S230 shown in FIG. 2, the above description applies equally, and a detailed description is omitted. FIG. 10 is an example of a cut-off value according to patient classification information according to an embodiment of the present disclosure.

[0179] According to one embodiment of the present disclosure, the processor (110) can generate a cut-off profile corresponding to a patient based on patient classification information (S920). Here, the patient classification information may include at least two of the patient's medical department, purpose of examination, patient age group, and patient symptoms.

[0180] The patient classification information according to one embodiment of the present disclosure may refer to information representing the clinical and demographic characteristics of the patient that may influence the determination of the cutoff value of the neural network model (500). Specifically, the patient's medical department may be information regarding the clinical department in which the patient is receiving treatment (e.g., cardiology, emergency medicine, family medicine, oncology, surgery, etc.), and the purpose of the examination may be information regarding the purpose of the examination to which the neural network model (500) is applied (e.g., screening, diagnostic, follow-up, pre-operative evaluation, etc.). Additionally, the patient age group may be information regarding the classification of the patient according to age (e.g., newborn, child, adolescent, adult, elderly, etc.), and the patient symptoms may be information regarding the clinical symptoms complained of by the patient (e.g., chest pain, shortness of breath, palpitations, fainting, etc.).

[0181] A cut-off profile according to one embodiment of the present disclosure may refer to a data structure in which a cut-off value is mapped to a specific combination of classification information. That is, the cut-off profile may be defined as a mapping relationship that takes a combination of patient classification information as input and outputs a cut-off value suitable for that combination of classification information.

[0182] A cut-off profile according to one embodiment of the present disclosure may be defined in the form of a multidimensional matrix in which each item of classification information, such as medical department, purpose of examination, patient age group, and patient symptoms, is a single dimension, and a cut-off value suitable for the combination of the corresponding classification information may be mapped to each cell of this multidimensional matrix. According to this multidimensional matrix structure, as items of classification information are added, the dimensions of the cut-off profile expand, and accordingly, the precision of determining the cut-off value can be improved.

[0183] Referring to FIG. 10, a cut-off profile according to one embodiment of the present disclosure may be represented in the form of a matrix (1000) in which different cut-off values ​​are mapped according to a combination of patient classification information. For example, the matrix (1000) may be configured with the medical department as the first dimension (1010), the purpose of the examination as the second dimension (1020), the patient age group as the third dimension (1030), and the patient symptoms as the fourth dimension (1040). Each cell of the matrix (1000) may have a cut-off value mapped to it that corresponds to the combination of classification information. For example, a low cut-off value (e.g., 0.30) may be mapped to a cell corresponding to the combination of classification information "Internal Medicine / Follow-up / Elderly / Symptomatic," and a high cut-off value (e.g., 0.50) may be mapped to a cell corresponding to the combination of classification information "Internal Medicine / Screening / Adult / Asymptomatic."

[0184] In addition, the recommended level of the cutoff value in the cutoff profile according to one embodiment of the present disclosure may be set differently depending on the clinical need profile as described in FIGS. 3 and 4. For example, a relatively low cutoff value may be mapped to a medical department belonging to a sensitivity-priority profile where high sensitivity is primarily required (e.g., internal medicine, oncology, emergency medicine, etc.) to minimize false negatives, and a relatively high cutoff value may be mapped to a medical department belonging to a specificity-priority profile where high specificity is primarily required (e.g., surgery, thoracic surgery, orthopedics, etc.) to minimize false positives. In addition, an intermediate level of cutoff value may be mapped to a medical department belonging to a balanced profile where sensitivity and specificity are required in balance (e.g., family medicine, neurology, etc.), and a relatively high cutoff value may be mapped to a medical department belonging to a profile in a relatively low prevalence environment (e.g., health checkup center, occupational and environmental medicine, etc.) to prevent unnecessary additional testing due to false positives.

[0185] A processor (110) can identify classification information of a patient obtained during the course of treatment and generate a cut-off profile corresponding to the patient based on the identified classification information of the patient. For example, the processor (110) can extract classification information such as the patient's medical department, purpose of examination, patient age group, and patient symptoms from the electronic medical records (EMR) of a medical institution, patient registration information, and input information of medical staff, and can generate a cut-off profile corresponding to the patient by combining the extracted classification information. For example, if a patient is a 65-year-old patient who visits the Department of Emergency Medicine complaining of chest pain and undergoing a screening test, the processor (110) can generate a cut-off profile corresponding to a combination of classification information such as "Department of Emergency Medicine / Purpose of Examination: Screening Test / Patient Age Group: Elderly / Patient Symptom: Chest Pain".

[0186] And, the processor (110) can identify a cut-off profile that matches the cut-off profile corresponding to the patient among a plurality of previously generated cut-off profiles in a medical environment (S930).

[0187] A plurality of previously generated cut-off profiles according to one embodiment of the present disclosure may refer to cut-off profiles that are generated in advance and stored in memory (120) or an external database in correspondence with each of the various combinations of patient classification information that may occur in a medical environment. That is, each of the plurality of previously generated cut-off profiles includes a cut-off value corresponding to a specific combination of classification information, and a suitable cut-off profile among them may be selected according to the patient classification information to which the neural network model (500) is applied in a medical environment.

[0188] A plurality of previously generated cut-off profiles according to one embodiment of the present disclosure may include two or more cut-off profiles corresponding to the same medical department. In particular, two or more cut-off profiles corresponding to the same medical department may be generated to have different cut-off values ​​corresponding to combinations of classification information in which at least one of the examination purpose, patient age group, and patient symptoms is different from each other.

[0189] For example, multiple cut-off profiles corresponding to the same emergency medicine department can be generated to have different cut-off values ​​when other classification information, such as the purpose of the examination, patient age group, and patient symptoms, differ even if they belong to the same medical department, such as a first cut-off profile (cut-off value 0.35) corresponding to a combination of classification information of "emergency medicine / screening test / elderly / chest pain" and a second cut-off profile (cut-off value 0.50) corresponding to a combination of classification information of "emergency medicine / confirmatory test / adult / respiratory distress." In this way, since different cut-off values ​​can be applied depending on the combination of patient classification information even within the same medical department, the neural network model (500) can apply a cut-off value suitable for the clinical characteristics of each patient, rather than applying a uniform cut-off value to patients belonging to the same medical department.

[0190] A plurality of previously generated cut-off profiles according to one embodiment of the present disclosure may be generated in advance based on reading data accumulated in a medical environment or operational data of another medical environment. For example, the processor (110) may generate cut-off profiles for each combination of classification information in advance by classifying reading data accumulated in a medical environment by combination of patient classification information and calculating a cut-off value suitable for the combination of classification information based on reading data corresponding to each combination of classification information. Alternatively, as described in FIGS. 3 and 4, cut-off profiles for each combination of classification information may be generated in advance based on cut-off values ​​applied for each combination of classification information in another medical environment included in a cluster corresponding to the medical environment.

[0191] The processor (110) can compare a cut-off profile corresponding to a patient with each of a plurality of previously generated cut-off profiles to identify a cut-off profile that matches the cut-off profile corresponding to the patient. Matching according to one embodiment of the present disclosure may mean that the combination of classification information of two cut-off profiles is identical or similar to a pre-set standard or greater.

[0192] For example, the processor (110) may identify a cut-off profile as a matching cut-off profile if the combination of classification information of a cut-off profile corresponding to a patient and the combination of classification information of each of the previously generated multiple cut-off profiles completely match. Alternatively, if the combination of classification information of a cut-off profile corresponding to a patient and the combination of classification information of each of the previously generated multiple cut-off profiles do not completely match, the processor (110) may calculate similarity by scoring the degree of matching for each item of classification information and identify the cut-off profile with the highest calculated similarity as a matching cut-off profile. At this time, the processor (110) may assign weights to each item of the patient's classification information. For example, similarity may be calculated by assigning the highest weight to the match of the medical department, and sequentially assigning lower weights to the match of the purpose of the examination, the patient's age group, and the patient's symptoms.

[0193] Alternatively, the processor (110) may convert each of the cut-off profiles corresponding to the patient and the previously generated multiple cut-off profiles into vectors, and identify a matching cut-off profile by calculating the similarity between the converted vectors.

[0194] Specifically, the processor (110) can convert each item of classification information included in the cut-off profile into a numeric value and represent it as a vector. For example, the processor (110) can convert categorical classification information (e.g., medical department, purpose of examination, patient symptoms, etc.) into a numeric vector using a one-hot encoding method or an embedding method, and can convert numeric or ordinal classification information (e.g., patient age, etc.) into a numeric vector by normalizing it. The processor (110) can combine the converted numeric vectors to generate an integrated vector corresponding to the cut-off profile.

[0195] In the case of transformation by an embedding method according to one embodiment of the present disclosure, a pre-trained embedding model may be used such that clinical similarity between each item of classification information is reflected in the spatial distance of the embedding vectors. For example, the embedding model may be trained so that embedding vectors for clinically similar medical departments (e.g., cardiology and emergency medicine, etc.) are located close to each other in coordinate space, and embedding vectors for clinically highly relevant patient symptoms (e.g., chest pain and dyspnea, etc.) are located close to each other in coordinate space.

[0196] The processor (110) can calculate the similarity between the integrated vector of the cut-off profile corresponding to the patient and the integrated vector of each of the previously generated multiple cut-off profiles. Here, the similarity between the vectors can be calculated using various distance measures, such as cosine similarity, Euclidean distance, Manhattan distance, and Mahalanobis distance. The processor (110) can identify the cut-off profile with the highest calculated similarity (or the closest distance) as the cut-off profile that matches the cut-off profile corresponding to the patient.

[0197] If there is no cut-off profile that matches the cut-off profile corresponding to the patient, the processor (110) may attempt to re-match based on only some items among the patient's classification information, or identify the default cut-off profile of the medical environment as the matching cut-off profile.

[0198] And, the processor (110) can determine the cutoff value corresponding to the identified cutoff profile as the cutoff value of the neural network model (500) (S940).

[0199] When multiple cutoff profiles are identified in which the similarity to the cutoff profile corresponding to the patient is greater than or equal to a preset standard, the processor (110) may determine the cutoff value of the neural network model (500) by weighting the cutoff value corresponding to each of the identified multiple cutoff profiles based on similarity. For example, when three cutoff profiles are identified in which the similarity to the cutoff profile corresponding to the patient is greater than or equal to a preset standard (e.g., 0.8), and their respective similarities are 0.95, 0.90, and 0.85 and their respective cutoff values ​​are 0.40, 0.42, and 0.45, the processor (110) may determine the cutoff value of the neural network model (500) by calculating the weighted average of the cutoff values ​​using similarity as a weight. According to this operation, the cutoff value can be determined by integrally reflecting information from multiple cutoff profiles similar to the cutoff profile corresponding to the patient, rather than relying solely on a single cutoff profile.

[0200] Meanwhile, if the similarity between the identified cut-off profile and the cut-off profile corresponding to the patient is less than a preset standard, the processor (110) may not apply the cut-off value corresponding to the identified cut-off profile as is, determine the default cut-off value of the medical environment as the cut-off value of the neural network model (500), or output a notification to the user indicating that the reliability of the matching result is low.

[0201] For example, if the neural network model (500) is a model that outputs a score corresponding to left ventricular systolic dysfunction from a patient's electrocardiogram data, different cut-off values ​​may be applied depending on the patient's classification information, even if it is the same neural network model (500). For example, for a 65-year-old patient who visited the Department of Emergency Medicine complaining of chest pain (for the purpose of screening patients suspected of left ventricular systolic dysfunction), a neural network model (500) with a relatively low cut-off value (e.g., 0.30) may be operated to minimize false negatives, and for a 40-year-old asymptomatic examinee who visited the Health Checkup Center (for the purpose of screening), a neural network model (500) with a relatively high cut-off value (e.g., 0.55) may be operated to prevent unnecessary additional testing due to false positives. Through this, even when the same neural network model (500) is operated in the same medical environment, a left ventricular systolic dysfunction diagnosis result with an optimal cut-off value applied according to the combination of each patient's classification information can be derived.

[0202] According to one embodiment of the present disclosure, the operation of determining a cutoff value based on a cutoff profile can be performed dynamically for each patient. That is, when the analysis of a patient's biometric data (10) is requested to the neural network model (500), the processor (110) can automatically recognize the classification information of the patient, generate a cutoff profile corresponding to the patient, identify a cutoff profile that matches the generated cutoff profile, and dynamically apply a cutoff value corresponding to the identified cutoff profile to the neural network model (500). Accordingly, the neural network model (500) can analyze the biometric data (10) with a cutoff value suitable for each patient's clinical characteristics applied, even in the same medical environment.

[0203] FIG. 11 is a block diagram of a computing device that dynamically sets the cut-off of a neural network model based on the characteristics of a medical environment according to another embodiment of the present disclosure.

[0204] Referring to FIG. 11, a computing device (1100) according to another embodiment of the present disclosure may include a processor (1110), memory (1120), a communication interface (1130), a display (1140), a user interface (1150), a sensing unit (1160), a camera (1170), a speaker (1180), and a microphone (1190). A detailed description of configurations shown in FIG. 11 that overlap with configurations shown in FIG. 1 will be omitted.

[0205] A computing device (1100) according to another embodiment of the present disclosure can directly acquire biometric data (10) of a patient (1) and input the acquired biometric data (10) into a neural network model (500) to determine the condition of the patient (1). In addition, the computing device (1100) can dynamically determine or update the cut-off value of the neural network model (500) based on the characteristics of the medical environment. To this end, the computing device (1100) can be implemented as various electronic devices such as a desktop, laptop, smartphone, tablet PC, smart watch, smart ring, or medical information terminal installed in a medical institution.

[0206] The display (1140) can display various images and text. Here, images include both still images and videos. The display (1140) can output a score output through the neural network model (500), a cutoff value applied to the neural network model (500), performance indicators of the neural network model (500), a judgment result regarding the condition of the patient (1), distribution information regarding the cutoff value of the medical environment, recommendation information regarding the update of the cutoff value, guide information, etc. Additionally, the display (1140) can visually output to a user (e.g., an administrator of a medical institution, medical staff, etc.) the cluster identification result in the coordinate space described in FIG. 4, the ROC curve described in FIG. 7, the user interface of the distribution information described in FIG. 8, and the matrix of the cutoff profile described in FIG. 10.

[0207] The display (1140) can be implemented as various types of displays such as LCD (Liquid Crystal Display Panel), OLED (Organic Light Emitting Diodes), LCoS (Liquid Crystal on Silicon), DLP (Digital Light Processing), etc. Additionally, the display (1140) may also include a driving circuit, a backlight unit, etc., which can be implemented in forms such as a-si TFT, LTPS (low temperature poly silicon) TFT, OTFT (organic TFT), etc.

[0208] Meanwhile, the display (1140) may be implemented as a touch screen by combining it with a touch panel. The display (1140) can perform the function of an output interface that outputs images through the touch screen, as well as an input interface that receives touch input from a user. Information regarding the characteristics of the medical environment (e.g., the size of the medical institution, the location of the medical institution, clinical division information, medical equipment information, etc.), target performance indicators of the neural network model (500), patient classification information, and update inputs of cut-off values ​​can be input through the display (1140).

[0209] The user interface (1150) is a configuration used by the computing device (1100) to perform interaction with the user, and may include at least one of a touch sensor, a motion sensor, a button, a jog dial, and a switch, but is not limited thereto. The computing device (1100) can receive information regarding the characteristics of the medical environment, target performance indicators of the neural network model (500), patient classification information, and user input regarding the update of the cut-off value through the user interface (1150). Based on the information input through the user interface (1150), the processor (1110) can determine or update the cut-off value of the neural network model (500) or derive a judgment result regarding the condition of the patient (1).

[0210] The sensing unit (1160) senses the patient's (1) biological signal to obtain the patient's (1) biological data (10). For example, the electrical signal of the patient's (1) heartbeat can be detected through a plurality of electrodes included in the sensing unit (1160) to obtain the patient's (1) electrocardiogram data. Alternatively, the sensing unit (1160) may include an optical sensor and obtain the patient's (1) photoplethysmogram (PPG) data through the optical sensor. The processor (1110) inputs the patient's (1) biological data (10) obtained directly through the sensing unit (1160) into a neural network model (500) to obtain a score corresponding to the likelihood of the patient's (1) specific disease, and can derive a judgment result regarding the patient's (1) condition by comparing the obtained score with a cut-off value determined based on the characteristics of the medical environment.

[0211] The camera (1170) can capture objects around the patient (1) to obtain an image of the patient (1). Based on the obtained image, the processor (1110) can identify part of the classification information of the patient (1) (e.g., the patient's age group, symptom information based on the patient's appearance, etc.). The camera (1170) can be implemented as an image sensor having a CMOS structure (CIS, CMOS Image Sensor), an image sensor having a CCD structure (Charge Coupled Device), etc. However, it is not limited thereto, and the camera (1170) can be implemented as a camera module of various resolutions capable of capturing a subject. Meanwhile, the camera (1170) can be implemented as a depth camera (e.g., IR depth camera, etc.), a stereo camera, or an RGB camera, etc.

[0212] The speaker (1180) is configured to output various audio data, for which various processing operations such as decoding, amplification, and noise filtering have been performed by an audio processing unit (not shown). The speaker (1180) can output various notification sounds or voice messages. According to one embodiment of the present disclosure, the processor (1110) can convert electrical signals into voice and output them through the speaker (1180). For example, the speaker (1180) can output in the form of voice messages a score obtained from the neural network model (500), a judgment result regarding the condition of the patient (1), information regarding a recommendation to update the cut-off value, a notification indicating that the performance indicator of the neural network model (500) has not met the target standard, and guide information recommending additional examination and specialist treatment.

[0213] A microphone (1190) can receive the user's voice. A processor (1110) can obtain the user's voice controlling the computing device (1100) through the microphone (1190) (e.g., input voice regarding characteristics of a medical environment, input voice regarding target performance indicators, voice regarding approval of updates to cutoff values, etc.). To this end, the computing device (1100) may include a model for voice recognition. Specifically, the computing device (1100) may include an Automatic Speech Recognition (ASR) model for converting the user's voice into text, and may also include a Natural Language Understanding (NLU) model.

[0214] In this way, a computing device (1100) according to another embodiment of the present disclosure can directly acquire biometric data (10) of a patient (1) through a sensing unit (1160), dynamically determine or update a cut-off value of a neural network model (500) based on the characteristics of a medical environment, and derive a judgment result regarding the condition of the patient (1) through a neural network model (500) to which the determined cut-off value is applied.

[0215] Additionally, the computing device (1100) can output information regarding the determination or update of cut-off values, guide information, etc., regarding the patient's (1) condition through a display (1140) or a speaker (1180), and can provide information about the patient (1) to the medical institution or be used as operational data for a neural network model (500) in another medical environment by transmitting it to an external computing device (e.g., a server of a medical institution, a computing device in another medical environment, etc.) through a communication interface (1130).

[0216] Meanwhile, according to one embodiment of the present disclosure, a non-transitory computer-readable medium may be provided that stores a program for sequentially performing a control method of a computing device that dynamically sets a cut-off of a neural network model based on the characteristics of the medical environment described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short moment, such as a register, cache, or memory. Specifically, the various applications or programs described above may be provided by being stored on a non-transitory computer-readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

[0217] The various embodiments of the present disclosure described above may be combined with additional embodiments and modified to the extent understandable to those skilled in the art in light of the detailed description above. The embodiments of the present disclosure are illustrative in all respects and should be understood as not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form. Accordingly, all modifications or variations derived from the meaning, scope, and equivalents of the claims of the present disclosure should be interpreted as being included within the scope of the present disclosure.

Claims

1. A method for controlling a computing device that dynamically sets a cutoff of a neural network model based on the characteristics of a medical environment, performed by a computing device comprising at least one processor, wherein A step of acquiring a neural network model trained to output a score corresponding to the patient's condition based on biometric data; A step of determining a cutoff value of the neural network model based on the characteristics of the medical environment; and A step of analyzing the patient's biological data based on the neural network model to which the above-determined cut-off value is applied to determine the judgment result regarding the patient's condition; comprising method.

2. In Paragraph 1, The step of determining the cutoff value of the above neural network model is, A step of identifying a cluster corresponding to the medical environment based on at least one of the size of the medical institution, the location of the medical institution, clinical division information, and the medical equipment information from which the biometric data was acquired, which are included in the characteristics of the medical environment; and The method includes the step of determining the cutoff value based on operational data of the neural network model in at least one other medical environment included in the cluster; The above operational data is, including cutoff values ​​applied to neural network models used in the aforementioned other medical environments, method.

3. In Paragraph 2, The step of identifying a cluster corresponding to the above medical environment is, A step of identifying a first candidate cluster based on the size of the medical institution for a plurality of different medical environments; A step of identifying a second candidate cluster based on the clinical division for the first candidate cluster; For the second candidate cluster above, a step of identifying a third candidate cluster based on the location of the medical institution; and With respect to the third candidate cluster, the method comprises the step of identifying a fourth candidate cluster based on the medical equipment that acquired the biometric data and determining it as a cluster corresponding to the medical environment. method.

4. In Paragraph 3, Information regarding the medical equipment that acquired the above biometric data is, It includes at least one of the manufacturer of the medical equipment, the model of the medical equipment, and the measurement protocol set on the medical equipment, The step of identifying the fourth candidate cluster is, Based on information regarding the medical equipment, the method comprises the step of identifying the fourth candidate cluster to include other similar medical environments where the similarity of the distribution of the score output from the neural network model is greater than or equal to a preset value. method.

5. In Paragraph 1, A step of calculating a performance indicator of the neural network model based on reading data of the medical environment accumulated over a preset period after the above-determined cutoff value is applied to the neural network model; and The method further includes the step of updating the cutoff value by comparing the above-calculated performance indicator with a preset target performance indicator; The reading data of the above medical environment is, A diagnosis result of the patient based on the score output from the above neural network model and a clinical confirmation result corresponding to the above diagnosis result, comprising method.

6. In Paragraph 5, The step of updating the above cutoff value is, If the above-calculated performance indicator falls short of the above-calculated target performance indicator, a step of repeatedly calculating the above-calculated performance indicator while adjusting the above-calculated cut-off value stepwise by a preset unit; and A step of updating to a cutoff value that first satisfies the above target performance indicator; comprising method.

7. In Paragraph 1, The step of determining the cutoff value of the above neural network model is, A step of receiving target performance indicators for the neural network model from the user; A step of obtaining a Receiver Operating Characteristic (ROC) curve for the above neural network model; and A step of determining the cutoff value that satisfies the input target performance indicator based on the above ROC curve; comprising method.

8. In Paragraph 7, The step of determining the cutoff value of the above neural network model is, A step of identifying a cluster corresponding to the medical environment based on at least one of the size of the medical institution, the location of the medical institution, clinical division information, and the medical equipment information from which the biometric data was acquired, which are included in the characteristics of the medical environment; and The method includes the step of determining the cutoff value based on operational data of the neural network model in at least one other medical environment included in the cluster; The step of obtaining the ROC curve for the above neural network model is, A step of obtaining an ROC curve corresponding to the cluster based on operational data of the neural network model in at least one other medical environment included in the identified cluster; method 9. In Paragraph 2, A step of obtaining anonymized operational data of a plurality of different medical environments included in the above-mentioned identified cluster; Based on the above-mentioned operational data, a step of calculating distribution information of cutoff values ​​and performance indicators used in the plurality of different medical environments; A step of outputting a user interface that displays the cut-off value determined in the medical environment and the performance indicators of the neural network model together with the distribution information to enable identification; and The method further comprises the step of updating the cutoff value determined in the medical environment based on user input received through the user interface. method.

10. In Paragraph 1, The step of determining the cutoff value of the above neural network model is, A step of generating a cut-off profile corresponding to the patient based on the classification information of the patient; A step of identifying a cut-off profile that matches a cut-off profile corresponding to the patient among a plurality of previously generated cut-off profiles in the medical environment; and The method includes the step of determining a cutoff value corresponding to the identified cutoff profile as the cutoff value of the neural network model; The classification information of the above patient is, Including at least two of the above patient's medical department, purpose of examination, patient age group, and patient symptoms, method.

11. In Paragraph 10, The above-mentioned multiple cut-off profiles are, It includes two or more cut-off profiles corresponding to the same medical department, and Two or more cut-off profiles corresponding to the same medical department mentioned above are, At least one of the above examination purpose, the above patient age group, and the above patient symptoms is generated to have a different cut-off value corresponding to a combination of different classification information, method.

12. A computing device that dynamically sets the cutoff of a neural network model based on the characteristics of a medical environment, Memory containing executable program code; and A neural network model that includes at least one core and, by executing the program code, is trained to output a score corresponding to the patient's condition based on biological data, determines a cutoff value of the neural network model based on the characteristics of the medical environment, and analyzes the patient's biological data based on the neural network model to which the determined cutoff value is applied to determine a judgment result regarding the patient's condition. Computing device.

13. A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs the operation of controlling a computing device that dynamically sets a cutoff of a neural network model based on the characteristics of a medical environment, and The above operation is, The operation of acquiring a neural network model trained to output a score corresponding to the patient's condition based on biometric data; An operation to determine a cutoff value of the neural network model based on the characteristics of the medical environment; and The operation of analyzing the patient's biological data based on the neural network model to which the above-determined cut-off value is applied to determine the judgment result regarding the patient's condition; comprising Computer program.