An improved method of, and apparatus for, medical data outcome analysis

Abstract

An Improved Method of, and Apparatus for, Medical Data Outcome Analysis There is provided a computer-implemented method for determining an uncertainty metric for a medical data abstraction process utilising a federated machine learning model executed on a computing node and having a plurality of personal abstraction models (PAMs) associated therewith, wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and trained using a local training dataset stored on the respective client node, wherein the method is executed by at least one hardware processor and comprises the steps of: inputting medical data relating to a patient medical record into the federated machine learning model to generate an outcome classification prediction for one or more medical parameters; inputting the medical data relating to the patient medical record into each trained PAM to generate an outcome classification prediction for the one or more medical parameters for each trained PAM; utilising the determined outcome classification predictions from each trained PAM in an ensemble voting process to determine an uncertainty metric of the outcome classification prediction in step a); d) determining whether the uncertainty metric meets one or more predefined criteria and, if so, accepting the determined outcome classification prediction and abstracting the patient medical record in an abstraction database.

Description

[0001] An Improved Method of, and Apparatus for, Medical Data Outcome Analysis

[0002] The present invention relates to an improved method of, and apparatus for, medical data outcome analysis. More particularly, the present invention relates to methods and apparatus for improving the accuracy and utility of abstracted medical data to improve outcome analysis using one or more trained machine learning models.

[0003] Increasingly, medical services are adopting electronic health record (EHR) systems. EHR systems enable patient medical records to be stored and retrieved digitally and provide opportunities for deeper analysis and extraction of clinical and / or medical data. Analysis of EHR records enables functions to be performed such as quality management, key performance indicator (kpi) management and medical research to be performed on large datasets.

[0004] However, whilst such endeavours may aim to improve patient care, the value of the analysis is dependent on the quality of data and the selection and interpretation of that data. Inaccurate and incomplete data can result in potential errors.

[0005] The process of selection and recording of relevant patient data from medical records in an EHR system is known as abstraction. To date, most EHR abstraction is performed manually. Such abstraction is known to have high error rates both in terms of the accuracy of the abstracted data and also the specific data which is abstracted from initial medical records.

[0006] For example, the significant volume of data resulting from available medical records and related information may mean that manual abstraction of the entirety of the data may be an intractable problem. Therefore, only a subset of the data may be abstracted manually, potentially providing an incomplete picture of the medical situation. Further, systematic and / or random errors by manual abstractors can lead to incorrect data analysis that may harm patient outcomes and research data.

[0007] The present invention aims, in embodiments, to address these issues.

[0008] The following introduces a selection of concepts in a simplified form in order to provide a foundational understanding of some aspects of the present disclosure. The following is not an extensive overview of the disclosure and is not intended to identify key or criticalelements of the disclosure or to delineate the scope of the disclosure. The following merely summarizes some of the concepts of the disclosure as a prelude to the more detailed description provided thereafter.

[0009] Several preferred aspects of the methods and systems according to the present invention are outlined below.

[0010] According to a first aspect of the present invention there is provided a computer-implemented method of determining an uncertainty metric for a medical data abstraction process utilising a federated machine learning model executed on a computing node and having a plurality of personal abstraction models (PAMs) associated therewith, wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and trained using a local training dataset stored on the respective client node, wherein the method is executed by at least one hardware processor and comprises the steps of: a) inputting medical data relating to a patient medical record into the federated machine learning model to generate an outcome classification prediction for one or more medical parameters; b) inputting the medical data relating to the patient medical record into each trained PAM to generate an outcome classification prediction for the one or more medical parameters for each trained PAM; c) utilising the determined outcome classification predictions from each trained PAM in an ensemble voting process to determine an uncertainty metric of the outcome classification prediction in step a); d) determining whether the uncertainty metric meets one or more predetermined criteria and, if so, accepting the determined outcome classification prediction and abstracting the patient medical record in an abstraction database.

[0011] In one embodiment, the one or more predetermined criteria may comprise meeting a predetermined threshold value for the uncertainty metric.

[0012] In one embodiment, a higher value of the uncertainty metric is indicative of a lower uncertainty and meeting the one or more predetermined criteria comprises meeting or exceeding a threshold value for the uncertainty metric.

[0013] In one embodiment, a lower value of the uncertainty metric is indicative of a lower uncertainty and meeting the one or more predetermined criteria comprises meeting or being below a threshold value for the uncertainty metric.In one embodiment, the one or more predefined criteria comprises a range of acceptable uncertainty values for the uncertainty metric.

[0014] In one embodiment, if the uncertainty metric does not meet the one or more predetermined criteria, the method further comprises: e) routing the patient medical record to a user for manual abstraction.

[0015] In one embodiment, the ensemble voting process generates the uncertainty metric based on the relative agreement or disagreement of the outcome classification predictions of the trained PAMs.

[0016] In one embodiment, the uncertainty metric is further qualified utilising statistical analysis of the outcome classification prediction of the federated machine learning model.

[0017] In one embodiment, the uncertainty metric is further qualified utilising statistical analysis of the outcome classification prediction of the federated machine learning model, selected from the group of: confidence intervals; calculated model entropy; softmax probabilities in connection with margins of confidence; and Monte Carlo dropout determination.

[0018] In one embodiment, the federated machine learning model comprises a plurality of machine learning sub-models. In one embodiment, the federated machine learning model comprises a plurality of chained machine learning sub-models for one or more medical parameters.

[0019] In one embodiment, the federated machine learning model comprises a plurality of independent parallel sets of machine learning sub-models each configured to generate outcome classification predictions for specific medical parameters. In one embodiment, each set may comprise a single machine learning sub-model or a plurality of chained machine learning sub-models.

[0020] In one embodiment, the one or more medical parameters comprise one or more of: a disease; an infection; a medical complication; a medical condition; a pharmaceutical complication; and a surgical complication. In one embodiment, an outcome classification prediction comprises a probability that the outcome of the relevant medical parameter has occurred.In one embodiment, the medical data comprises electronic health record (EHR) data. In one embodiment, the EHR data comprises one or more of: free medical text; medical notes; medical reports; chart notes; laboratory values; vital signs; diagnoses; and procedure codes.

[0021] In one embodiment, the federated machine learning model executed on the computing node comprises a global federated machine learning model. In one embodiment, the global federated machine learning model is a trained global federated machine learning model and wherein each PAM is further trained using a local training dataset stored on the respective client node, the local training dataset comprising abstraction outcomes selected by a user in relation to one or more medical parameters for a plurality of patient medical records.

[0022] In one embodiment, the computing node comprises a central computing node in a centralised federated learning environment.

[0023] In one embodiment, the computing node comprises a further client computing node in a decentralised federated learning environment.

[0024] According to a second aspect of the present invention, there is provided a computer system comprising a computing node and a plurality of client computing nodes, the computing note being configured to execute a federated machine learning model thereon, the federated machine learning model being configured to perform data abstraction on input medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters and further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the computer system is operable to carry out the method of the first aspect.

[0025] According to a third aspect of the present invention, there is provided a computer program comprising instructions which, when executed by a computer system, cause the computer to perform the steps of the first aspect.

[0026] According to a fourth aspect of the present invention, there is provided a computer-implemented method of medical data abstraction performance benchmarking utilising afederated machine learning model [executed on a central computing node] and configured to perform data abstraction on input medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters, the federated machine learning model further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the method is executed by at least one hardware processor and comprises the steps of: a) training a PAM on a client computing node using a local training dataset to generate a trained PAM, the local training dataset comprising abstraction outcomes selected by a user in relation to one or more medical parameters for a plurality of patient medical records; b) transferring the trained PAM to a benchmarking computing environment; c) performing, using the trained PAM, data abstraction on a benchmark input medical dataset to generate an outcome classification prediction in relation to one or more medical parameters; d) comparing the outcome classification prediction from the trained PAM with a true outcome classification prediction to determine a performance metric for the user associated with the trained PAM.

[0027] In one embodiment, the federated machine learning model comprises a plurality of machine learning sub-models. In one embodiment, the federated machine learning model comprises a plurality of chained machine learning sub-models for one or more medical parameters.

[0028] In one embodiment, the federated machine learning model comprises a plurality of independent parallel sets of machine learning sub-models each configured to generate outcome classification predictions for specific medical parameters.

[0029] In one embodiment, each set may comprise a single machine learning sub-model or a plurality of chained machine learning sub-models.

[0030] In one embodiment, when the federated machine learning model is configured to generate outcome classification predictions for one or more specific medical parameters associated with a particular set of sub-model(s), the remaining sets are not activated. In one embodiment, the performance metric comprises an accuracy metric or error rate metric.

[0031] In one embodiment, the one or more medical parameters comprise one or more of: a disease; an infection; a medical complication; a medical condition; a pharmaceuticalcomplication; and a surgical complication. In one embodiment, an outcome classification prediction comprises a probability that the outcome of the relevant medical parameter has occurred.

[0032] In one embodiment, the medical data comprises electronic health record (EHR) data. In one embodiment, the EHR data comprises one or more of: free medical text; medical notes; medical reports; chart notes; laboratory values; vital signs; diagnoses; and procedure codes.

[0033] In one embodiment, the federated machine learning model executed on the computing node comprises a global federated machine learning model. In one embodiment, the benchmarking computing environment is executed on the computing node.

[0034] In one embodiment, the computing node comprises a central computing node in a centralised federated learning environment.

[0035] In one embodiment, the computing node comprises a further client computing node in a decentralised federated learning environment.

[0036] According to a fifth aspect of the present invention, there is provided a computer system comprising a computing node and a plurality of client computing nodes, the computing note being configured to execute a federated machine learning model thereon, the federated machine learning model being configured to perform data abstraction on input medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters and further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the computer system is operable to carry out the method of the fourth aspect.

[0037] According to a sixth aspect of the present invention, there is provided a computer program comprising instructions which, when executed by a computer system, cause the computer to perform the steps of the fourth aspect.According to a seventh aspect of the present invention, there is provided a computer-implemented method of training a federated machine learning model executed on a computing node and configured to perform data abstraction on input medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters, the federated machine learning model further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the method is executed by at least one hardware processor and comprises the steps of: a) receiving, at a client computing node, medical data relating to a patient medical record; b) inputting the medical data relating to the patient medical record into the PAM associated with the client computing node to generate one or more outcome classification predictions for one or more medical parameters; c) displaying, on a graphical user interface associated with the client computing node, the outcome classification predictions for the one or more medical parameters and at least a part of the medical data associated with the patient medical record; d) receiving one or more user input actions to select and / or deselect the one or more outcome classification predictions for the one or more medical parameters; e) determining user parameter data representative of the user’s interaction with or analysis of the medical data of the patient medical record; f) updating the PAM utilising the selected and / or deselected outcome classification predictions and determined user parameters to generate an updated PAM.

[0038] In one embodiment, an abstraction session for abstraction of the patient medical record is initiated at step c) and terminated following selection and / or deselection of all available outcome classification predictions for one or more medical parameters in step d).

[0039] In one embodiment, the user parameter data comprises one or more selected portions of the medical data of the patient medical record that the user has been determined to have focussed on during the abstraction session.

[0040] In one embodiment, step e) comprises determining the time period for which different portions of the medical data of the patient medical record are visible on the graphical user interface during the abstraction session.In one embodiment, one or more portions of the medical data of the patient medical record which were visible on the graphical user interface for longest time during the abstraction session are selected as the one or more selected portions of the medical data.

[0041] In one embodiment, step e) comprises determining the portion of the medical data of the patient medical record are visible on the graphical user interface when the user selected or deselected one or more outcome classification predictions to determine the one or more selected portions of the medical data.

[0042] In one embodiment, for any selected outcome classification predictions in step d), the method further comprises: g) generating an abstracted medical record outcome and storing the abstracted medical record outcome in an abstraction database.

[0043] In one embodiment, the method further comprises: h) identifying, on the graphical user interface, one or more elements of the medical data associated with the determination of the outcome classification predictions for the one or more medical parameters.

[0044] In one embodiment, the step of identifying one or more elements of the medical data in step h) comprises highlighting the one or more elements of the medical data on the graphical user interface.

[0045] In one embodiment, the highlighting comprises displaying the one or more elements in a different colour, font, location and / or format from the elements of the medical data having no or a lesser association with the determination of the outcome classification predictions for the one or more medical parameters.

[0046] In one embodiment, steps a) to f) are repeated for subsequent patient medical records.

[0047] In one embodiment, the federated machine learning model comprises a plurality of machine learning sub-models. In one embodiment, the federated machine learning model comprises a plurality of chained machine learning sub-models for one or more medical parameters.

[0048] In one embodiment, the federated machine learning model comprises a plurality of independent parallel sets of machine learning sub-models each configured to generate outcome classification predictions for specific medical parameters. In one embodiment,each set may comprise a single machine learning sub-model or a plurality of chained machine learning sub-models.

[0049] In one embodiment, when the federated machine learning model is configured to generate outcome classification predictions for one or more specific medical parameters associated with a particular set of sub-model(s), the remaining sets are not activated.

[0050] In one embodiment, the one or more medical parameters comprise one or more of: a disease; an infection; a medical complication; a medical condition; a pharmaceutical complication; and a surgical complication.

[0051] In one embodiment, an outcome classification prediction comprises a probability that the outcome of the relevant medical parameter has occurred.

[0052] In one embodiment, the medical data comprises electronic health record (EHR) data.

[0053] In one embodiment, the EHR data comprises one or more of: free medical text; medical notes; medical reports; chart notes; laboratory values; vital signs; diagnoses; and procedure codes.

[0054] In one embodiment, the federated machine learning model executed on the computing node comprises a global federated machine learning model. In one embodiment, the global federated machine learning model is a trained global federated machine learning model and wherein each PAM is configured to be further trained using the local training dataset stored on the respective client node.

[0055] In one embodiment, the computing node comprises a central computing node in a centralised federated learning environment.

[0056] In one embodiment, the computing node comprises a further client computing node in a decentralised federated learning environment.

[0057] According to an eighth aspect of the present invention, there is provided a computer system comprising a computing node and a plurality of client computing nodes, the computing note being configured to execute a federated machine learning model thereon, the federated machine learning model being configured to perform data abstraction oninput medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters and further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the computer system is operable to carry out the method of the seventh aspect.

[0058] According to a ninth aspect of the present invention, there is provided a computer program comprising instructions which, when executed by a computer system, cause the computer to perform the steps of the seventh aspect.

[0059] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which:

[0060] Figure 1 shows a schematic diagram of a medical analysis computing system according to an embodiment;

[0061] Figure 2 shows a schematic diagram of an abstraction system according to an embodiment;

[0062] Figure 3 shows a schematic view of a graphical user interface according to an embodiment;

[0063] Figure 4 shows a more detailed view of the graphical user interface of Figure 3;

[0064] Figure 5 shows a flow chart of a method according to an embodiment;

[0065] Figure 6 shows a flow chart of a method according to a further embodiment; and

[0066] Figure 7 shows a flow chart of a method according to a further embodiment.

[0067] Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numbers are used to identify like elements illustrated in one or more of the figures, whereinshowings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same.

[0068] Various examples and embodiments of the present disclosure will now be described. The following description provides specific details for a thorough understanding and enabling description of these examples. One of ordinary skill in the relevant art will understand, however, that one or more embodiments described herein may be practiced without many of these details.

[0069] Likewise, one skilled in the relevant art will also understand that one or more embodiments of the present disclosure can include other features and / or functions not described in detail herein. Additionally, some well-known structures or functions may not be shown or described in detail below, so as to avoid unnecessarily obscuring the relevant description.

[0070] Medical computing system configuration

[0071] The present invention relates to a medical analysis computing system 1 . Figure 1 shows a computing system 1 according to an embodiment. The computing system 1 may be local to a hospital, clinic or other medical establishment, or may be remote therefrom.

[0072] The computing system 1 comprises a computing node in the form of computing device 10. Computing device 10 has one or more physical processors 12, a computer-readable physical memory 14 and a non-transitory storage device 16 such as a hard disk drive or solid-state drive.

[0073] The computing device 10 may take any suitable form and may comprise, for example, a cloud-based computing system, a remote computer connected over a network, a local workstation, a server system or a dedicated medical system console.

[0074] The one or more physical processors 12 may comprise any suitable processor types; for example, central processing units (CPUs), graphical processing units (GPUs) or any other suitable processor such as FPGAs and stream processors.

[0075] The storage device 16 may take any suitable form and may include storage device(s) 16 local to the computing device 10 and / or storage devices external to the computing device10. For example, the storage device 16 may comprise cloud or other virtual networked storage.

[0076] The computing device 10 further comprises an interface 18 through which medical data is received. The interface 18 may take any suitable form and may be in the form of a communications interface and / or a network connection to a suitable source of input medical data.

[0077] A computing application 20 is run on the computing system and is operable to cause the computing device 10 to perform the method of the present invention. The computing application 20 comprises an analysis system 100 according to an embodiment.

[0078] The computing system 1 further comprises a network 30. The network 30 may take any suitable form and may be a local network (for example, a Wi-Fi or Ethernet network), a cloud network (through internet or cellular communication connections) or a mixed network utilizing both technologies. The network 30, therefore, places no specific location constraints on the components of the computing device 10.

[0079] The computing device 10 is connected to a number of networked systems forming part of the computing system 1. The computing system 1 further comprises an abstraction database 40 for storage of processed data. The abstraction database 40 is shown in Figure 1 connected across the network 30 but this need not be so and the abstraction database 40 may be local to the computing device 10.

[0080] The computing system 1 further comprises an electronic health record (EHR) system 50 and a plurality of client nodes in the form of client devices 60-1, 60-2.. 60-n. The EHR system 50 and client devices are connected to the computing device 10 across the network 30.

[0081] The EHR system 50 comprises an interface through which an operator can enter data, a database of medical records and one or more registries. The EHR system 50 is configured to enable the collection and use of individual patient health-related information and has facilities to enable results management, order entry and support, and decision support. The database comprises health information and other data relating to patient details.A registry may comprise an organised system operable to utilise observational study methods to collect uniform clinical data to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves one or more predetermined scientific, clinical, or policy purposes. In order for the registry to be effective and useful, the EHR data must be abstracted.

[0082] The client devices 60-1...60-n are connected to the network 30 and to the computing device 10 and provide interfaces through which human data abstractors (HDAs) can perform abstraction on received EHR data. This is known as “human in the loop” (HITL) abstraction.

[0083] The client devices 60 may take any suitable form; for example, desktop computers, laptop computers, mobile telephones, smartphones, tablet computers or other computing devices. Each client device 60 comprises at least one hardware processor and a memory.

[0084] The HDAs may enter abstracted data using a suitable graphical user interface (GUI) run on the client devices 60 as described later in relation to Figures 3 and 4. This may take any suitable form. For example, in embodiments, this may comprise one or more defined Application Programming Interfaces (APIs) run on the client devices 60. Each API may have a user interface specific to the particular client device 60 (e.g. smartphone, tablet, PC etc.) that is being used. The user interface provides a portal through which the user can interact with the computing system 1 and, specifically, data from the computing device 10 and EHR 50. In embodiments, data requests to these elements from the client devices 60 may be handled by the API of each client device 60.

[0085] The components and operation of the computing system 1 and computing device 10 will be discussed in further detail below with reference to Figure 2. It is noted here that while various components and operations have been described herein in terms of “modules”, “units” or “components,” these terms are not limited to single units or functions. Moreover, functionality attributed to some of the modules or components described herein may be combined and attributed to fewer modules or components.

[0086] In addition, the specific location where components and operations of the present invention are performed is not material. For example, components may be located on one or more of the client devices 60, computing device 10 or elsewhere and operations may be performed locally on the client devices 60 or on the computing device 10 as required.Figure 2 shows an analysis system 100 according to an embodiment of the present invention. In embodiments, the analysis system 100 forms part of the computing application 20 run on the computing device 10 although this need not be the case.

[0087] In embodiments, the analysis system 100 comprises an abstraction module 102 and an abstraction database 102D. The abstraction module 102 comprises a data processor 104, a federated abstraction model (FAM) 106 and a case router 108. The FAM 106 comprises a plurality of personal abstraction models (PAMs) 106-1 ...106-n as described below.

[0088] The abstraction module 102 is configured to receive input medical data from the EHR module 50 in the form of patient records or cases. This may involve any health- or medical-related data in relation to one or more patients. For example, this may include data relating to operations or medical care that a patient has received. Non-limiting examples may include: a medical diagnosis; a surgical procedure; administration of a drug; therapeutic treatments; and / or other information relating to the treatment, care or diagnosis of a pathology related to the patient.

[0089] The form this input medical data may take may vary. In embodiments, it may comprise written or electronic documentation such as free medical text, medical notes and / or reports, or it may take other forms of digital records such voice recordings and inputs. In embodiments, written or electronic documentation may comprise one or more of the following non-limiting examples: chart notes; lab values; vital signs; diagnoses; and procedure codes.

[0090] In embodiments, voice data may comprise voice inputs captured from sources such as automated or manual telephone interviews with patients, voice signals captured from ambient recordings of healthcare provider / patient conversations or voice notes made by a medical practitioner or the patient. In embodiments, this data may be transcribed or otherwise recorded in text format (e.g., voice to text translation).

[0091] The data may also include time stamps in relation to particular time points for patient care or outcomes. For example, a patient’s EHR record may comprise data from the time of case logging to a relevant follow-up time point (e.g. 30 or 90 days following a medical event, such as 30 or 90 days following a surgical procedure or one year following the administration of a given drug of interest).The abstraction module 102 is configured to receive electronic medical data in real-time or substantially real-time from the EHR system 50 and is configured to capture relevant health care events and outcomes from this data as described below. In other words, the abstraction module 102 is configured to abstract outcomes from received EHR data to capture outcomes registered in structured data (e.g., written or electronic records) and to determine outcomes from non-structured data sources (e.g., voice data).

[0092] The components of the abstraction module 102 will now be described.

[0093] Data processor 104

[0094] The data processor 104 is operable to pre-process the incoming medical data prior to analysis. The data processor 104 may reformat and / or harmonise the data into a predetermined, unified format for use by downstream components of the abstraction system 102. In other words, the output of the data processor 104 is formatted patient medical data suitable for use in downstream components of the abstraction system 102.

[0095] Federated abstraction model 106

[0096] The federated abstraction model (FAM) 106 is operable to receive the formatted medical data from the data processor 104 and perform processing and analysis thereon.

[0097] The FAM 106 comprises one or more machine learning models operable to abstract the received medical data for one or more outcomes of interest. In embodiments, the FAM 106 comprises one or more trained machine learning models. In embodiments, the FAM 106 may comprise a single machine learning model, multiple parallel machine learning models, multiple chained machine learning models or any suitable combinations of the above.

[0098] By “chained” machine learning model is meant a series of at least two machine learning models where the data to be processed is input into a first machine learning model, and the output from the first machine learning model is then used as the input to a second machine learning model. The output from the second machine learning model is then used as the generated solution.In embodiments, the FAM 106 may comprise a suitable array of machine learning models grouped according to abstraction task. In embodiments, the FAM 106 may comprise one or more models configured for particular or specific abstraction tasks. In embodiments, an abstraction task may comprise identifying data relating to a particular medical parameter of interest. A medical parameter of interest may comprise one or more of a medical condition, a medical trend or a medical procedure of interest.

[0099] For example, one or more machine learning models may be configured to abstract data to identify a medical parameter in the form of instances of deep vein thrombosis (DVT). A particular form of model structure (e.g. a single machine learning model type or a chain of machine learning models) may be optimised for abstraction of data relating to evidence or recordals of DVT instances in EHR data. In that form, the FAM 106 engages this machine learning model or set of machine learning models for the abstraction task.

[0100] In embodiments, the FAM 106 may be configured for a single abstraction task relating to abstraction of data relating to a single medical parameter such as the example described above. Thus, the abstraction module 102 may be configured to abstract only data related to this single medical parameter (e.g., DVT).

[0101] Alternatively, the FAM 106 may be configured for multiple abstraction tasks. In embodiments, this may involve using a single machine learning model configuration (e.g., a single machine learning model type or a chain of machine learning models of similar or different types) for multiple abstraction tasks to identify data relating to a plurality of medical parameters (e.g., DVT and other medical parameters such as surgical site infections(SSIs).

[0102] Alternatively or additionally, the FAM 106 may comprise separate parallel groups of machine learning models each optimised for a single abstraction task. For example, a first machine learning model group may comprise a single machine learning model or a chain of machine learning models which is optimised for a first abstraction task and a second machine learning model group comprising a single machine learning model or a chain of machine learning models which is optimised for a second abstraction task.

[0103] By way of non-limiting example, the first abstraction task may be in relation to a first medical parameter such as DVT instance identification and the second abstraction task may be in relation to a second medical parameter such as SSI identification.If the FAM 106 comprises multiple machine learning model groups each specialised for a different specific abstraction task, then only the machine learning model(s) in the machine learning model group relevant to that specific abstraction task will be utilised in the abstraction process. In other words, if the FAM 106 comprises a plurality of parallel machine learning model groups, only a subset of those groups will be used for a specific abstraction task, with the other groups not being engaged in the task.

[0104] By providing such an arrangement, the FAM 106 can be optimised for multiple abstraction tasks independently to achieve the best abstraction performance for each abstraction task in relation to each medical parameter of interest. The specific machine learning models forming part of the FAM 106 may be selected as appropriate for a given task.

[0105] Machine learning model configurations which may be used may include: Light Gradient Boosting Machines (LGBM) models; Long Short Term Memory (LSTM) models; support vector machines (SVMs); random decision forest models; transformer architectures; and large language models (LLMs).

[0106] Gradient boosting comprises a machine learning technique which has application in regression and classification problems. A strong prediction model is derived which comprises an ensemble of weak prediction models such as decision trees. A step-wise process may be used to generate the model through, for example, steepest descent minimization.

[0107] LSTM is an artificial recurrent neural network architecture comprising feedback as well as feedforward connections. An LSTM unit may comprise a cell, an input gate, an output gate and a forget gate. The cell retains values over an arbitrary time interval when the flow of information into and out of the cell is regulated by the gates.

[0108] An SVM comprises a non-probabilistic binary linear classifier which utilises a set of training examples in one or more categories and generates a model that assigns new examples to a particular category.

[0109] Random decision forests comprise ensemble machine learning methods which operate by constructing a multitude of decision trees during a training process and outputting the classthat is the mode of the classes (classification) or mean / average prediction (regression) of the individual trees.

[0110] Transformer architectures operate by conversion of text into tokenised numerical representations, each token then being converted into a vector utilising a word embedding table. Each token may then be contextualised using a parallel multi-head attention mechanism, enabling relevant tokens to be assigned greater importance and less relevant tokens lesser importance. Transformer architectures may take the form of Bi-directional encoders from transformers (BERT) architectures. BERT is a language model using an encoder-only transformer architecture and a form of LLM. If an LLM is used it may comprise known models such as GPT-4 or Llama.

[0111] By providing the model structure of the present invention, the abstraction process can be optimised for particular abstraction tasks. For example, the inventors have found that for a relatively straightforward abstraction task such as determining whether a patient suffered a deep vein thrombosis, approaches such as Light Gradient Boosting Machines (LGBM) and Long Short Term Memory (LSTM) models have shown some success. For more complex abstraction problems such as surgical site infections, the inventors have found that BERT or GPT type architectures may be preferred.

[0112] In embodiments, the preferred selection of machine learning model or models is a function of a model structure that is least cumbersome to train but which still provides acceptable performance. The model selected is which again depends on the problem at hand and the relative complexity.

[0113] In embodiments, model chaining may be utilised to achieve specific technical benefits. For example, as opposed to a traditional chain structure where the output of a first model is unconditionally input into a second, the chaining may be used only if certain conditions are met.

[0114] For example, consider a first machine learning model which has a particular abstraction task of determining whether a surgical site infection (SSI) is present. If the first model determines a high probability of an SSI in the input data, only then is the output from the first model input into the second model. The advantage is that the second model will only receive cases where a SSI is highly likely, and so the output assessment from the second model is more accurate.In embodiments, such a two model chain may comprise a first model in the form of a BERT and a second model such as GPT. In such an embodiment, the GPT may be prompted to function as an expert data abstractor, and based on the definitions for a surgical site infection from the given registry, determine whether an SSI is present. The chaining of models thus increase the accuracy of the outcome classification predictions overall.

[0115] The FAM 106 comprises a federated model structure. A federated abstraction model utilises federated learning principles. In a federated learning model according to an embodiment, a specific task (in this case, data abstraction) is addressed by a group of participating client nodes (client devices 60-1 ...60-n) under the coordination of a central computing node (the central computing device 10). In embodiments, the computing device 10 has a central global model. Each client device 60-1, 60-n has a local training dataset which remains on the client device 60 and is not uploaded to the central computing node (computing device 10).

[0116] This approach is known as centralised federated learning. However, this is not to be taken as limiting and other federated learning configurations may be utilised with the present invention. For example, in embodiments, a decentralised federated model structure may be utilised. In such a structure, coordination of the global model updates is achieved by the client computing nodes (client devices 60) and so the central computing node is not required. Thus, in such a configuration, the global FAM 106 may not be stored on the computing device 10 and, instead, may be stored on or accessed by one or more of the client computing nodes (client devices 60).

[0117] In use, each client device 60 has a personal abstraction model (PAM) 106P which computes an update to the current global model on the computing device 10 and this update alone is transferred between the client devices 60 and the computing device 10. The updates are specific to the improvement of the current model of the PAM 106P. As a result, the model training does not require direct access to the raw data. This makes federated learning a good choice for situations where confidentiality and privacy are of key importance, such as when dealing with health records.

[0118] The FAM 106 may comprise any suitable sort of federated learning model. The FAM 106 may use any suitable federation strategy, for example federated averaging (FedAvg). In a FedAvg model, a local model is trained by each client device using the client device’s owndata. Parameters are allocated to a large number of clients and then recycling, training, and updating the local models is performed by the computing device 10 without any information from the client devices being shared. Model parameters across all client devices are then aggregated to obtain an average. The client devices are then updated with the global model and the process continues iteratively until a convergence is reached.

[0119] An alternative strategy is Federated Adam, based on the Adam optimiser. In this approach, training data is distributed among client devices and a moving average of the model update follows a momentum decay rule. The weights of the model are updated by the moving averages and a learning rate for each weight is adjusted. An exponential moving average of the first moment of the gradient is computed and the weights are adapted based on the estimated variance and mean gradient.

[0120] A further alternative strategy is Federated Yogi (FedYogi). In this methodology, the client devices are used to train a model without sharing data from the client devices. A single global model is trained and is used by all client devices.

[0121] A further alternative strategy is Federated Adaptive Gradients (FedAdagrad) in which a gradient descent optimization algorithm is utilised. Gradients for the variable are optimised and the algorithm is based on adapting the learning rate and first-order information with at least some of the properties of second-order methods.

[0122] Further alternative strategies are Federated Proximity, which is a derivative of FedAvg. This strategy may be used to reduce statistical heterogeneity and utilises terms that regularise each local training loss. The local model and its deviation are also controlled by this model.

[0123] Finally, Stochastic Controlled Averaging for Federated Learning (Scaffold) is a technique where a training process is coordinated across different nodes across the network. This approach reduces the need for multiple rounds of communication and resolves the issue of heterogeneity. A server-side learning rate is employed to regulate variates and reduce control variates.

[0124] Any suitable federation strategy, including the examples above, may be used with the present invention. However, in embodiments, whilst the PAMs 106P are federated instances of the FAM 106 and updates may be used to update the global machine learningmodel of the FAM 106, the PAMs 106P may not be updated with each iterative cycle, or may only be updated infrequently as described below in model training.

[0125] The FAM 106 may, as noted above, comprise an array of machine learning models or groups of machine learning models configured for one or more abstraction tasks. In embodiments, one or more of these models is federated and associated with a corresponding PAM 106P on the client devices 60-n. Updates for each federated model are computed on the client devices 60-n as described below.

[0126] The FAM 106 may be trained in any appropriate manner. In embodiments, the FAM 106 may be trained to capture postoperative complications based on analysis of free medical text in combination with structured input medical data (for example, laboratory data values or vital signs).

[0127] In embodiments, the FAM 106 receives formatted medical input data and is operable to generate an output comprising an abstracted dataset. In embodiments, the FAM 106 assigns the output abstracted dataset a probability that each of the outcome classifications of interest are present in the output dataset.

[0128] FAM uncertainty estimate

[0129] In addition, the FAM 106 is operable to generate an associated uncertainty estimate for the outcome classification predictions. The uncertainty estimate may be in the form of an uncertainty metric. The uncertainty metric may be a measure of the likely accuracy of the outcome classification predictions and determines the next steps for the generated abstracted output data.

[0130] In embodiments, the uncertainty estimate may be determined from methods and inputs such as descriptive statistics of how often the outcome in question is present for the given case type (e.g., how often a specific operation on average is complicated by a surgical site infection).

[0131] Alternatively or additionally, in embodiments, the uncertainty estimate can be based on statistical methods such as entropy / perplexity and probability distributions. Statistical methods may comprise one or more of: confidence intervals; calculated model entropy;softmax probabilities in connection with margins of confidence; and Monte Carlo dropout determination.

[0132] In embodiments, the present invention may generate or refine an uncertainty estimate based on the method of surrogate benchmarking as described below. In this scenario,, different available Personal Abstraction Models (PAMs) from all client devices 60-n where the system is deployed are also deployed on the case and perform the outcome classification predictions. These PAM-based outcome classification predictions are then assembled in an ensemble voting system. In embodiments, the agreement or disagreement of these models forms the basis of an uncertainty estimate. In other words, the ensemble voting system uses a hard voting type where the final decision is based on the majority class determined by the PAMs.

[0133] This is a unique aspect of the present invention where a plurality of personal abstraction models (i.e. a plurality of unique, remotely-trained versions of the FAM 106) are used in an ensemble voting system to determine an uncertainty value. This will be described below in relation to the method aspects.

[0134] Case router 108

[0135] The case router 108 is configured to route the abstracted case from the FAM 106 to the abstraction database 102D or to one of the client devices 60 for HITL abstraction.

[0136] The case router 108 determines the appropriate pathway based on one or more predefined criteria for the uncertainty metric. In embodiments, a predefined criterion may involve a threshold uncertainty being met. By “being met” is meant that the uncertainty metric meets or exceeds a threshold uncertainty (where a higher value of an uncertainty metric is indicative of a lower uncertainty) or meets or is below a threshold uncertainty (where a lower value of an uncertainty metric is indicative of a lower uncertainty). If used, a threshold uncertainty may be specific to one or more conditions or parameters and may take values based on the uncertainty mechanism(s) used. The uncertainty metric may take a range of values and a predefined range or subrange may comprise a predefined criterion.

[0137] Uncertainty may also be situation dependent and may be calibrated as the system is operated. In embodiments, feedback from each route from the case router 108 may be used to inform uncertainty thresholds. In embodiments where the uncertainty threshold isbased on surrogate benchmarking PAMs, the threshold may be the level of agreement between PAMs.

[0138] In embodiments, statistical or ensemble methods may be used to generate a percentage uncertainty. Any suitable threshold may be established for this (for example, a 25% uncertainty threshold). The threshold may be medical parameter dependent - for example, in cases where errors in abstraction may have more serious consequences (e.g., in the cases of life threatening conditions, outbreaks of disease or infection in a particular location), the uncertainty threshold may be lower.

[0139] In embodiments, if the threshold is not met, the classification is deemed to be too uncertain and have the case is sent to the client devices 60 for HITL abstraction.

[0140] In embodiments, the uncertainty thresholds may be based on a combination of the ensemble PAM voting and statistical or empirical principles. For example, a final uncertainty estimate may comprise components of uncertainty derived from the PAM voting ensembles and the uncertainty derived from measures such as model entropy.

[0141] A specific threshold for determining the route (either HITL or direct to the abstraction database 102D) may be calibrated as the system is operated in use to ensure optimal performance. Determination of optimal performance may, in embodiments, comprise analysis related to sensitivity, specificity, and positive and negative predictive values. A suitable threshold would ensure that these values remain above an acceptable cut-off (for example, greater than 95%).

[0142] This may be calibrated and controlled by selecting a number of cases (potentially selected at random) for direct HITL abstraction to serve as gold standards for internal calibration and control purposes.

[0143] If the threshold uncertainty or predetermined uncertainty level is met, then the abstracted data is deemed to be sufficiently accurate that human analysis is not needed and the automatically-abstracted data (using the FAM 106) can be sent directly to the abstraction database 102D. This workflow is referred to as “Stand Alone Model” (SAM) abstraction.Alternatively, if the uncertainty does not meet the predetermined threshold (i.e., the uncertainty is deemed to be unacceptably high), the case router 108 is configured to forward the abstraction case for HITL abstraction.

[0144] Load balancer 110

[0145] The load balancer 110 is configured to select the most appropriate client device 60 to send the next abstraction case to. This may be done on the basis of any appropriate load balancing methodology to ensure fair and efficient distribution of the incoming workload to the HDAs. In embodiments, the load balancer 110 assigns a given abstraction case to the HDA with the fewest pending abstraction cases. This may be based on a task list of pending abstraction cases associated with each client device 60 used by a HDA.

[0146] Client devices 60, personal abstraction models 106P and HITL

[0147] Once forwarded to a particular client device 60 based on a task list for the client device 60 associated with a HDA, the HDA employs an abstraction platform to review the EHR data and perform manual abstraction of the medical data contained therein.

[0148] To assist each HDA, each client device 60 is associated with a personal abstraction model (PAM) 106P. Each PAM 106P comprises a local machine learning model associated with the global model of the FAM 106. In other words, each PAM 106P comprises an instance of the federated machine learning model (FAM 106) local to and executed on a respective client computing node in the form of client device 60.

[0149] Consequently, each PAM 106P comprises the same one or more machine learning models operable to abstract the received medical data for one or more outcomes of interest as the FAM 106. As a result, like the FAM 106, each PAM 106P may comprise a single machine learning model, multiple parallel machine learning models, multiple chained machine learning models or any suitable combinations of the above.

[0150] Each PAM 106P is configured to be trained using a local training dataset stored on the respective client node. In embodiments, at initialisation of the PAM 106P, the PAM 106P has a predefined degree of training in order to function as an abstraction assistant. This is discussed in more detail in the training section below.In general, the one or more machine learning models of the FAM 106 may be trained on an appropriate initial dataset prior to initialisation. Consequently, the PAM 106P, as an instance of the initial trained machine learning models of the FAM 106 local to the client computing device 60, will also have been trained on the initial dataset.

[0151] Personal abstraction models 106P, graphical user interface 150 and dynamic attention adjustment

[0152] Figures 3 and 4 show views of a GUI 150 of the abstraction platform of a client device 60. The GUI 150 provides a portal through which a user of the client device 60 (i.e. the HDA) can interact with the computing system 1 and, specifically, the PAM 106P and EHR data.

[0153] Figure 3 shows a high level schematic of an exemplary GUI 150. The GUI 150 comprises three main information fields 152, 154, 156. Field 152 comprises patient personal details for each EHR medical record such as name, date of birth, address, nationality and other personal details. These details enable the medical record to be associated with the patient. However, in embodiments, the field 152 does not form part of the data used in the abstraction process.

[0154] Information field 154 comprises the medical data of the EHR medical record. In embodiments, this comprises text data. The text data may involve any health- or medical-related data in relation to the patient. For example, this may include data relating to operations or medical care that a patient has received. Non-limiting examples may include: a medical diagnosis; a surgical procedure; administration of a drug; therapeutic treatments; and / or other information relating to the treatment, care or diagnosis of a pathology related to the patient.

[0155] The form this medical data may take may vary. In embodiments, it may comprise written or electronic documentation such as free medical text, medical notes and / or reports, or it may take other forms of digital records such voice to text converted or transcribed voice recordings and inputs. In embodiments, the text may comprise written or electronic documentation may comprise one or more of the following non-limiting examples: chart notes; lab values; vital signs; diagnoses; and procedure codes.

[0156] Information field 154 comprises a window in which the text of the medical data of the EHR medical record can be viewed by the HDA. However, in general the text of the medicaldata will extend beyond the window size such that the HDA may need to scroll up and down through the text (i.e., move the text up and down within the window of information field 154) in order to review the medical data and arrive at an abstraction outcome.

[0157] Finally, information field 156 relates to an outcome classification prediction of the PAM 106P and associated options. This will be described below in relation to Figure 4.

[0158] Figure 4 shows a more detailed version of a part of the GU1 150 shown in Figure 3. Figure 4 shows just the information fields 154, 156. Information field 156 relates to the outcome classification prediction of the PAM 106P. Three data elements 156A, 156B and 156C are shown.

[0159] Data element 156A comprises an outcome classification prediction for one or more medical parameters. Data element 156B shows the probability of those classification predictions for the respective medical parameters and data element 156C provides interactable action buttons operable to receive user input to select (i.e., accept or approve) or deselect (i.e., decline or refute) the outcome classification predictions of the PAM 106P.

[0160] Returning to information field 154, in embodiments, the PAM 106P may draw attention to specific regions of the GUI or particular data elements of the text of the abstraction case. In embodiments, each PAM 106P is operable to show the HDA how the outcome classification prediction was derived from the text of the medical record displayed in information field 154.

[0161] In embodiments, this comprises identifying, on the graphical user interface 150, one or more elements of the medical data in information field 154 associated with the determination of the outcome classification predictions for the one or more medical parameters.

[0162] This may be done by any suitable method, for example, by highlighting passages and places in the EHR data that supports the outcome classification predictions of the FAM 106.

[0163] As such, each PAM 106P can highlight EHR medical text indicative of the outcome classification prediction of interest. This is shown in Figure 4 where highlighted elements 154H are shown in the window of the information field 154. In other words, the highlighted elements in the PAM 106P may draw the HDA’s attention to this part of the EHR chart text.Whilst the highlighting shown in Figure 4 comprises changing the background colour of highlighted text, other approaches may be used in the alternative or additionally. In embodiments, the highlighting may comprise displaying the one or more elements in a different colour, font, location, and / or format from the elements of the medical data having no or a lesser association with the determination of the outcome classification predictions for the one or more medical parameters.

[0164] The highlighting described above shows the HDA how the PAM 106P arrived at a particular probability of a particular outcome classification prediction shown in window 156. However, the inventors of the present invention have recognised that the user’s (HDA's) response to the data displayed in the GU1 150 can be used by the abstraction module 102 to improve the training of the PAM 106P.

[0165] In other words, with each abstracted case completed by a HDA, the PAM 106P associated with the client device 60 of that HDA is operable to learn the workflows and decisions of the HDA and improve the processing of cases thereby.

[0166] For example, in embodiments, the PAM 106P for a given client device 60 associated with a particular HDA may be operable to determine the sections of the EHR medical data in the window 154 which was in context when a given outcome classification prediction was declared or refuted by the HDA. This data input enables dynamic attention adjustment processing whereby the PAM 106P can, in embodiments, be further trained to improve analysis and / or show the HDA specific parts of the EHR data for a given case in a particular order or configuration.

[0167] This may be done in any suitable manner. For example, in embodiments, the outcome classification predictions for the one or more medical parameters may be displayed in information field 156 and at least a part of the medical data associated with the patient medical record displayed in information field 154.

[0168] Commonly, the volume of medical data will mean that only a part of the medical data can be viewed in the information field 154 at any one time. This means that the user is required to scroll back and forth through the medical data to obtain the confirmation or validation required to select or deselect one or more outcome classification predictions shown in data fields 156A, 156B and 156C in information field 156.Therefore, in embodiments, the user’s focus can be determined by analysing the portion of the medical data that the user is viewing at any one time in the information field 154. This enables the determination of user parameter data representative of the user’s interaction with or analysis of the medical data of the patient medical record. To determine this it is useful to define an abstraction session.

[0169] Whilst an abstraction session may take any suitable form and time, in embodiments an abstraction session for abstraction of the patient medical record may be defined as being initiated when the medical data is displayed on the GUI 150 with the outcome prediction classifications, and terminated following selection and / or deselection of all available outcome classification predictions for one or more medical parameters by user actions to select the buttons operable to receive user input in data field 156C.

[0170] In embodiments, the user parameter data may then be defined as comprising one or more selected portions of the medical data of the patient medical record that the user has been determined to have focussed on during the abstraction session. This data and association with the outcome classification predictions and the user’s acceptance or rejection thereof can be used to update the PAM 106P to improve outcome classification prediction and the user’s workflow for subsequent abstraction sessions.

[0171] In embodiments, the user parameter data may be obtained in any suitable manner. For example, the time period for which different portions of the medical data of the patient medical record are visible on the graphical user interface 150 during the abstraction session may be determined to identify the portions which the user spent the most time analysing or reading and these portions may be utilised as the user parameter data.

[0172] Alternatively or additionally, the user parameter data may be obtained by identifying the portion of the medical data of the patient medical record which were visible on the graphical user interface when the user selected or deselected one or more outcome classification predictions to determine the one or more selected portions of the medical data.

[0173] Following this determination, the PAM 106P may be updated and the process repeated.

[0174] This has several unique benefits. For example, particular data may be displayed in a particular order, location, highlighting or overlay to draw the HDA’s attention rapidly torelevant sections of the EHR data indicative of a given outcome. As a result, the dynamic attention adjustment thus enables the PAM 106P associated with a client device 60 of a particular HDA to improve abstraction accuracy and also to learn the HDA’s particular or favourite workflow, and adapt the presentation of data for subsequent abstraction cases accordingly.

[0175] Personal abstraction models 106P and surrogate benchmarking

[0176] The nature of the FAM 106 in which the local models (the PAMs 106P) are trained locally on each client device 60 provides for unique assessment methodologies.

[0177] Conventionally, benchmarking of the performance (e.g., accuracy, reliability) of each HAD is challenging because of the confidential nature of EHR medical data. As a result, conventional comparison of a HDA's outcome analysis results with a benchmark or “gold standard” outcome analysis is not possible because this would potentially breach patient confidentiality or data privacy given the sensitive nature of the data in question.

[0178] However, the present invention, in embodiments, provides a unique opportunity for benchmarking. As each HDA processes EHR abstraction cases on their client device 60 alongside an associated PAM 106P, the PAM 106P is trained based on the responses of the HDA to each abstraction case. Over time, the local PAM 106P associated with a client device 60 of a particular HDA becomes a reflection of the HDA's own performance.

[0179] In embodiments, it is possible to utilise the local PAM 106P in an independent and confidential assessment of the respective HDA's performance. In embodiments, this is done by transferring the respective local PAM 106P to a centralised environment having a dataset with predetermined characteristics. In embodiments, the dataset may comprise verified and / or determined high accuracy “gold-standard” abstractions.

[0180] When deployed on this gold-standard dataset, the variance of the individual PAM 106P from the gold standard then becomes an effective surrogate for the individual HDA’s abstraction performance.

[0181] In other words, an independent assessment of the abstraction performance (for example, accuracy and / or error rate) of the HDA can be assessed without the need to use confidential and / or private medical data.The methodology may generate a suitable metric of performance. If the performance metric does not meet a predefined level or standard, then an alarm or notification may be generated. The alarm or notification may indicate that the HDA under assessment may require appropriate retraining.

[0182] Further, the surrogate benchmarking feature has further technical advantages. Once a certain number of abstraction cases have been successfully processed to ensure that each local PAM 106P is trained sufficiently to function as a representative surrogate of the HDA using the client device 60 with which the PAM 106P is associated, that PAM 106P can function as a proxy HDA without human involvement.

[0183] This may be useful in, for example, deployment of one or more local PAMs 106P on the input medical data from the EHR 50 and data processor 102 to assist with the uncertainty estimations which forms the basis of the operation of the case router 108.

[0184] In such a scenario, given the PAMs 106P are deemed to be effective surrogates for each individual HDA from all systems where the FAM 106 is operating, each input medical case can be assessed in surrogate by all available PAMs 106P and thus, in principle, by all HDA’s using system without human involvement. This has the technical effect of improving the accuracy of the uncertainty estimations without increasing the burden on HDAs.

[0185] Data routing following HITL abstraction and outcome monitor 112

[0186] Once the HDA has made an abstraction assessment for each abstraction case, the abstracted case is sent to the abstraction database 102D. In other words, both the HITL and SAM approach results in abstracted data being reposited to the abstraction database 102D.

[0187] Data in the abstraction database 102D is monitored by an outcome monitor 112 for data correlations or numbers of similar incidences. For example, if correlations or number of instances of a particular outcome exceed a predetermined threshold or metric, then an alert may be raised.For example, an alert may be raised to notify clinical personnel if the incidences of an outcome of interest (e.g., surgical site infections) rise above a pre-specified threshold or in a particular location.

[0188] Model training

[0189] The FAM 106 may be trained in any appropriate manner. In embodiments, the FAM 106 may be trained to capture postoperative complications based on analysis of free medical text in combination with structured input medical data (for example, laboratory data values or vital signs).

[0190] Upon initialisation of a new system, the FAM 106 may be trained on a suitable existing dataset of input medical data. The form this training medical data may take may vary. In embodiments, it may comprise written or electronic documentation such as free medical text, medical notes and / or reports, or it may take other forms of digital records such voice recordings and inputs. In embodiments, written or electronic documentation may comprise one or more of the following non-limiting examples: chart notes; lab values; vital signs; diagnoses; and procedure codes. The training dataset may comprise outcome classifications from one or more human abstractors and may therefore represent “true” outcomes.

[0191] Once trained, the PAMs 106P may be created as instances of the initially trained FAM 106.

[0192] As described above, the PAMs 106P are then each trained locally by the abstraction of medical records by the respective HDA using the respective client device 60 that the PAM 106P is executed on as described above. As a result, whilst each PAM 106P may start out as a federated instance of the FAM 106, the PAMs 106P become personalised over time through operation on abstraction cases alongside the HDAs.

[0193] Conventionally, federated models undergo an iterative training cycle in which updates from the federated client nodes are then used to update collectively the global model. In embodiments, the further trained PAMs 106P may be configured to send model updates to the global machine learning model(s) of the FAM 106 for update. This process may be carried out on any suitable timeframe; for example, days, weeks, months or longer. In embodiments, this period may be 3 months to 1 year.Conventionally, it is known that at the end of a learning round, an updated global model may update the federated models to render them instances of the updated global model before a new round of training begins.

[0194] However, in embodiments of the present invention, it may be preferred to maintain the PAMs 106P as personalised models unique to the user (i.e., HDA). As a result, in embodiments, the PAMs 106P may send model updates to the FAM 106 but a corresponding update from the global model to the PAMs 106P may not occur on every training cycle. However, there may be instances where a PAM 106P is required to be updated; for example, for use with a new or replacement HDA, a new instance of a PAM 106P for a new user (i.e., HDA) or where the performance of an HDA is determined to be inadequate and a new model may be required to restart the process. In addition, global updates of the PAMs 106P may be required on a predetermined timeframe as required.

[0195] METHOD

[0196] The following method relates to the steps occurring during operation of the outcome determination method using the medical analysis computing device 10. Figure 5 shows a flow chart of the method according to an embodiment.

[0197] In embodiments, the computer-implemented method of Figure 5 comprises determining an uncertainty metric for a medical data abstraction process utilising a federated machine learning model (FAM 106P) executed on a computing node and having a plurality of PAMs 106P associated therewith, wherein each PAM 106P comprises an instance of the federated machine learning model local to and executed on a client computing node and trained using a local training dataset stored on the respective client node.

[0198] Step 200: Log case in EHR

[0199] At step 200, the process begins with the logging of a case in the EHR 50. This may involve the input of patient-related health data in one or more formats or types of media. For example, the input EHR data may comprise

[0200] The abstraction module 102 is configured to receive input medical data from the EHR module 50 in the form of patient records or cases. This may involve any health- or medical-related data in relation to one or more patients. For example, this may include data relatingto operations or medical care that a patient has received. Non-limiting examples may include: a medical diagnosis; a surgical procedure; administration of a drug; therapeutic treatments; and / or other information relating to the treatment, care or diagnosis of a pathology related to the patient.

[0201] The form this input medical data may take may vary. In embodiments, it may comprise written or electronic documentation such as free medical text, medical notes and / or reports, or it may take other forms of digital records such voice recordings and inputs. In embodiments, written or electronic documentation may comprise one or more of the following non-limiting examples: chart notes; lab values; vital signs; diagnoses; and procedure codes.

[0202] In specific embodiments, electronic health record data may comprise one or more of the following: - medical chart notes; lab values; vital signs; diagnoses codes; procedure codes; radiology reports; microbiology reports; procedure information (e.g. surgery; radiology); transfer information (ward, intensive care, emergency department etc); discharge dispositions (to home, other hospital, patient has died etc); and medication information (both pre-hospital and administered at the hospital).

[0203] In embodiments, voice data may comprise voice inputs captured from sources such as automated or manual telephone interviews with patients, voice signals captured from ambient recordings of healthcare provider / patient conversations or voice notes made by a medical practitioner or the patient.

[0204] The data may also include time stamps in relation to particular time points for patient care or outcomes. For example, a patient’s EHR record may comprise data from the time of case logging to a relevant follow-up time point (e.g. 30 or 90 days following a medical event, such as 30 or 90 days following a surgical procedure or one year following the administration of a given drug of interest).

[0205] Therefore, if the logging of a new record or data element for an existing patient is carried out, this may be correlated or connected with earlier recorded data for that patient. The method proceeds to step 202.

[0206] Step 202: Transfer new abstraction case to abstraction module 102At step 202, the new EHR record to be abstracted is transferred to the abstraction module 102. The electronic medical data is received in real-time or substantially real-time from the EHR system 50 for subsequent processing to capture relevant health care events and outcomes from this data as described below.

[0207] In other words, the aim of the process is to abstract outcomes from received EHR data to capture outcomes registered in structured data (e.g., written or electronic records) and to determine outcomes from non-structured data sources (e.g., voice data). The method proceeds to step 204.

[0208] Step 204: Process incoming data

[0209] At step 204, the data processor 104 pre-processes the incoming medical data prior to analysis. The data processor 104 reformats and / or harmonises the data into a predetermined, unified format for use by downstream components of the abstraction system 102. In other words, the output of the data processor 104 in step 204 is formatted patient medical data suitable for use in downstream components of the abstraction system 102.

[0210] The method proceeds to step 206.

[0211] Step 206: Input data into federated abstraction model

[0212] At step 206, the formatted data is inputted into the FAM 106 which is operable to perform processing and analysis thereon.

[0213] The FAM 106 comprises a machine learning model operable to abstract the received medical data for one or more outcomes of interest. In embodiments, the FAM 106 comprises a trained machine learning model.

[0214] A federated abstraction model utilises federated learning principles. In a federated learning model according to an embodiment, a specific task (in this case, data abstraction) is addressed by a group of participating client devices 60-1 ...60-n under the coordination of a central computing device 10. The computing device 10 has a central global model. Each client device 60-1 , 60-n has a local training dataset which remains on the client device 60 and is not uploaded to the computing device 10. The method proceeds to step 208.Step 208: Process data in federated abstraction model and generate output

[0215] In embodiments, the FAM 106 receives formatted medical input data and is operable to generate an output comprising an abstracted dataset. In embodiments, the FAM 106 assigns the output abstracted dataset a probability that each of the outcome classifications of interest are present in the output dataset.

[0216] Step 210: Generate uncertainty estimate

[0217] Step 210 may occur concurrently with step 208. At step 210, the FAM 106 generates an associated uncertainty estimate for the outcome classification predictions in step 208. The uncertainty estimate may be in the form of an uncertainty metric. The uncertainty metric may be a measure of the likely accuracy of the outcome classification predictions and determines the next steps for the generated abstracted output data.

[0218] In embodiments, the uncertainty estimate may be determined from methods and inputs such as descriptive statistics of how often the outcome in question is present for the given case type (e.g., how often a specific operation on average is complicated by a surgical site infection).

[0219] Alternatively or additionally, in embodiments, the uncertainty estimate can be based on statistical methods such as entropy / perplexity and probability distributions. Statistical methods may comprise one or more of: confidence intervals; calculated model entropy; softmax probabilities in connection with margins of confidence; and Monte Carlo dropout determination.

[0220] In embodiments, the present invention may generate or refine an uncertainty estimate based on the method of surrogate benchmarking. In this scenario, different available Personal Abstraction Models (PAMs) from all client devices 60-n where the system is deployed are deployed on the medical record and perform the outcome classification predictions.

[0221] These PAM-based outcome classification predictions are then assembled in an ensemble voting system. In embodiments, the agreement or disagreement of these models forms the basis of an uncertainty estimate. In specific embodiments, the ensemble voting systemuses a hard voting type where the final decision is based on the majority class determined by the PAMs.

[0222] In other words, this step may involve the process of inputting the medical data relating to the patient medical record into each trained PAM 106P to generate an outcome classification prediction for the one or more medical parameters for each trained PAM 106P. Then, the determined outcome classification predictions from each trained PAM 106P are utilised in an ensemble voting process to determine an uncertainty metric of the outcome classification prediction made by the global machine learning model(s) of the FAM 106.

[0223] This is a unique aspect of the present invention where a plurality of personal abstraction models PAM 106P-n (i.e., a plurality of unique, remotely- and personally-trained versions of the FAM 106) are used in an ensemble voting system to determine an uncertainty value.

[0224] In embodiments, the agreement or disagreement of these models forms the basis of the uncertainty estimate which is generated in step 210.

[0225] Step 212: Route case

[0226] At step 212, the case router 108 routes the abstracted case from the FAM 106 direct to the abstraction database 102D or to one of the client devices 60 for HITL abstraction.

[0227] The case router 108 determines the appropriate pathway based on one or more predefined criteria for the uncertainty metric. In embodiments, a predefined criterion may involve a threshold uncertainty being met. By “being met” is meant that the uncertainty metric meets or exceeds a threshold uncertainty (where a higher value of an uncertainty metric is indicative of a lower uncertainty) or meets or is below a threshold uncertainty (where a lower value of an uncertainty metric is indicative of a lower uncertainty). If used, a threshold uncertainty may be specific to one or more conditions or parameters and may take values based on the uncertainty mechanism(s) used. The uncertainty metric may take a range of values and a predefined range or subrange may comprise a predefined criterion.

[0228] In other words, it is determined whether the uncertainty metric meets a predetermined threshold and, if so, accepting the determined outcome classification prediction and abstracting the patient medical record in an abstraction database.Uncertainty may also be situation dependent and may be calibrated as the system is operated. In embodiments, feedback from each route from the case router 108 may be used to inform uncertainty thresholds. In embodiments where the uncertainty threshold is based on surrogate benchmarking PAMs 106P, the threshold may be the level of agreement between PAMs 106P (i.e. total agreement, more than a certain number or proportion of PAMs 106P being in agreement etc.).

[0229] In embodiments, statistical or ensemble methods may be used to generate a percentage uncertainty. Any suitable threshold may be established for this (for example, a 25% uncertainty threshold). The threshold may be medical parameter dependent - for example, in cases where errors in abstraction may have more serious consequences (e.g., in the cases of life threatening conditions, outbreaks of disease or infection in a particular location), the uncertainty threshold may be lower.

[0230] In embodiments, if the threshold is not met, the classification is deemed to be too uncertain and have the case is sent to the client devices 60 for HITL abstraction.

[0231] In embodiments, the uncertainty thresholds may be based on a combination of the ensemble PAM voting and statistical or empirical principles. For example, a final uncertainty estimate may comprise components of uncertainty derived from the PAM voting ensembles and the uncertainty derived from measures such as model entropy.

[0232] A specific threshold for determining the route (either HITL or direct to the abstraction database 102D) may be calibrated as the system is operated in use to ensure optimal performance. Determination of optimal performance may, in embodiments, comprise analysis related to sensitivity, specificity, and positive and negative predictive values. A suitable threshold would ensure that these values remain above an acceptable cut-off (for example, greater than 95%).

[0233] This may be calibrated and controlled by selecting a number of cases (potentially selected at random) for direct HITL abstraction to serve as gold standards for internal calibration and control purposes.If the threshold uncertainty or predetermined uncertainty level is met, then the abstracted data is deemed to be sufficiently accurate that human analysis is not needed and the method proceeds to step 214.

[0234] Alternatively, if the uncertainty does not meet the predetermined threshold (i.e. the uncertainty is deemed to be unacceptably high), the case router 108 forwards the case for HITL and the method proceeds to step 220.

[0235] Step 214: Store case in abstraction database 102D

[0236] At step 214, the abstracted data is stored in the abstraction database. This may comprise storing of the automatically-abstracted data (using the FAM 106) in step 212 (a workflow referred to as “Stand Alone Model” (SAM) abstraction) or it may be from step 228 following HITL abstraction. In other words, both the HITL and SAM approach results in abstracted data being reposited to the abstraction database 102D.

[0237] The abstracted outcome data may comprise any suitable medical outcome that is to be monitored. The nature of this will depend on the abstraction use case or registry.

[0238] In specific embodiments, technically useful abstracted outcomes may be as set out in the US National Surgical Quality Improvement Program (NSQIP) which captures 30 day postoperative outcome data on postoperative complications such as infections, thrombosis, organ failure, sepsis, reoperations, readmissions etc.

[0239] Obtaining abstracted outcome data in relation to the above conditions and parameters is of significant clinical importance and use. The method proceeds to step 216.

[0240] Step 216: Review data and determine correlations

[0241] At step 216, data in the abstraction database 102D is monitored by the outcome monitor 112 for data correlations or numbers of similar incidences. For example, if correlations or number of instances of a particular outcome exceed a predetermined threshold or metric, then an alert may be raised in step 218.

[0242] Step 218: Generate alertIf, at step 216, a correlation is determined between recorded instances, an alert may be raised. For example, an alert may be raised to notify clinical personnel if the incidences of an outcome of interest (e.g., surgical site infections) rise above a pre-specified threshold or in a particular location.

[0243] Step 220: Load balance

[0244] At step 220, the load balancer 110 selects the most appropriate client device 60 to send the next abstraction case to. This may be done on the basis of any appropriate load balancing methodology to ensure fair and efficient distribution of the incoming workload to the HDAs. In embodiments, the load balancer 110 assigns a given abstraction case to the HDA with the fewest pending abstraction cases. This may be based on a task list of pending abstraction cases associated with each client device 60 used by a HDA. Once a client device 60 is selected, the method proceeds to step 222.

[0245] Step 222: Data abstracted by HDA

[0246] At step 222, the HDA employs the abstraction platform to review the EHR data and perform manual abstraction of the medical data contained therein.

[0247] To assist each HDA, each client device 60 is associated with a personal abstraction model (PAM) 106P. Each PAM 106P comprises a local abstraction model associated with the global model of the FAM 106.

[0248] Each PAM 106P is operable to show the respective HDA the prediction that a given class of outcome is present in the text of the abstraction case. In addition, in embodiments, the PAM 106P may draw attention to specific regions of the GUI or particular data elements of the abstraction case.

[0249] This may be done by any suitable method, for example, by highlighting passages and places in the EHR data that supports the predictions of the FAM 106. As such, each PAM 106P can highlight EHR text indicative of data of interest. For example, for matters such as a surgical site infection, the PAM 106P may draw the HDA’s attention to this part of the EHR chart text. Once the data has been abstracted, the method proceeds to step 214.FURTHER METHOD EMBODIMENT

[0250] The following method relates to the steps occurring during operation of a dynamic attention module using the medical analysis computing device 10. Figure 6 shows a flow chart of the method according to an embodiment.

[0251] Step 300: Receive new abstraction case

[0252] At step 300, the load balancer 110 has the most appropriate client device 60 and sent an abstraction case thereto. This is received by the client device 60 and readied for abstraction by an HDA who is a user of the client device 60. In other words, medical data relating to a patient medical record is received at a client computing node for further analysis. The method proceeds to step 302.

[0253] Step 302: Perform outcome classification prediction

[0254] At step 302, the medical data relating to the patient medical record is inputted into the PAM 106P associated with the client computing node to generate one or more outcome classification predictions for one or more medical parameters.

[0255] To assist each HDA, each client device 60 is associated with a personal abstraction model (PAM) 106P. Each PAM 106P comprises a local abstraction model associated with the global model of the FAM 106. In this step, the PAM 106P generates an outcome classification prediction that a given outcome is present in the text of the abstraction case.

[0256] Step 304: Display outcome classification predictions and medical data

[0257] At step 304, the GU1 150 is utilised to display the outcome classification predictions for the one or more medical parameters determined in step 302 together with at least a part of the medical data associated with the patient medical record. At this step it may be determined that an abstraction session has been initiated for the patient medical record.

[0258] The outcome classification predictions are displayed in information field 156. Within this field, data element 156A comprises an outcome classification prediction for one or more medical parameters. Data element 156B shows the probability of those classification predictions for the respective medical parameters and data element 156C providesinteractable action buttons operable to receive user input to select (i.e. , accept or approve) or deselect (i.e., decline or refute) the outcome classification predictions of the PAM 106P.

[0259] Meanwhile, information field 154 comprises a window in which the text of the medical data of the EHR medical record can be viewed by the HDA. However, in general the text of the medical data will extend beyond the window size such that the HDA may need to scroll up and down through the text (i.e., move the text up and down within the window of information field 154) in order to review the medical data and arrive at an abstraction outcome.

[0260] Step 306: Highlight data

[0261] Step 306 may be optional. In embodiments, the PAM 106P may draw attention to specific regions of the GUI or particular data elements of the text of the abstraction case. In embodiments, in this step, each PAM 106P is operable to show the HDA how the outcome classification prediction was derived from the text of the medical record displayed in information field 154.

[0262] In embodiments, this comprises identifying, on the graphical user interface 150, one or more elements of the medical data in information field 154 associated with the determination of the outcome classification predictions for the one or more medical parameters.

[0263] This may be done by any suitable method, for example, by highlighting passages and places in the EHR data that supports the classification predictions of the FAM 106. As such, each PAM 106P can highlight EHR medical text indicative of the outcome classification prediction of interest. This is shown in Figure 4 where highlighted elements 154H are shown in the window of the information field 154. In other words, the highlighted elements in the PAM 106P may draw the HDA’s attention to this part of the EHR chart text.

[0264] Whilst the highlighting shown in Figure 4 comprises changing the background colour of highlighted text, other approaches may be used in the alternative or additionally. In embodiments, the highlighting may comprise displaying the one or more elements in a different colour, font, location, and / or format from the elements of the medical data having no or a lesser association with the determination of the outcome classification predictions for the one or more medical parameters.Step 308: Review and select data

[0265] At step 308, the HDA employs the abstraction platform to review the EHR data and perform manual abstraction of the medical data contained therein.

[0266] Each case for abstraction comprises multiple information elements that in combination align with the definition of the abstract outcome under consideration. Consider the example of abstraction of a case to identify a surgical site infection. In the US national surgical quality improvement program (NSQIP), a superficial surgical site infection is defined as follows:

[0267] 1 . Occurs within 30 days of an operative procedure

[0268] 2. Involves the skin or subcutaneous tissue

[0269] 3. At least one of the following criteria are met:

[0270] a. Purulent discharge from the wound

[0271] b. Pathogens are isolated from a wound culture or fluid

[0272] c. The incision is deliberately opened by the surgeon and the patient exhibits at least one of the following symptoms:

[0273] i. Pain or tenderness

[0274] ii. Localized swelling

[0275] ill. Redness

[0276] iv. Heat

[0277] v. If a culture is obtained from the wound with pathogenic growth, this criterion is omitted

[0278] 4. The surgeon diagnoses a superficial site infection and documents this in the EHR chart .

[0279] Therefore, for this abstracted outcome, there are multiple pathways for diagnosis according to the NSQIP criteria.

[0280] The user interface shows the HDA that the PAM 106P considers there to be a percentage (for example, 85%) likelihood of a surgical site infection, and further shows that the PAM 106P bases this assessment on what could be the following factors:

[0281] 1 . A physician chart note entry stating that there is localized swelling (highlighting the specific part of the text describing this);

[0282] 2. A microbiology record indicating that a pathogenic bacteria was isolated from the wound culture;3. The surgeon writes a chart entry stating that there is suspicion of a wound infection.

[0283] The user may take some time to review the medical data and arrive at a selection. When ready, the user may make one or more user input actions on the GUI 150 to select and / or deselect the one or more outcome classification predictions for the one or more medical parameters.

[0284] Once the user has made all the necessary selections, the abstraction session may be determined to have terminated for the particular medical record. The method proceeds to step 310.

[0285] Step 310: Record selected elements to obtain user parameter data

[0286] At step 304, the PAM 106P for the given client device 60 associated with a particular HDA determines the section (data element, field, or entry) of the EHR data which was in context when a given outcome was declared or refuted by the HDA. In other words, user parameter data representative of the user’s interaction with or analysis of the medical data of the patient medical record is determined in this step.

[0287] Step 310 may occur simultaneously with steps 304 to 308 and may run alongside. In other words, recording of user actions during the abstraction session is determined or recorded in this section to determine the user parameter data.

[0288] In embodiments, the user parameter data may then be defined as comprising one or more selected portions of the medical data of the patient medical record that the user has been determined to have focussed on during the abstraction session. This data and association with the outcome classification predictions and the user’s acceptance or rejection thereof in step 308 can be used to update the PAM 106P to improve outcome classification prediction and the user’s workflow for subsequent abstraction sessions.

[0289] In embodiments, the user parameter data may be obtained in any suitable manner. For example, the time period for which different portions of the medical data of the patient medical record are visible on the graphical user interface 150 during the abstraction session may be determined to identify the portions which the user spent the most time analysing or reading and these portions may be utilised as the user parameter data.Alternatively or additionally, the user parameter data may be obtained by identifying the portion of the medical data of the patient medical record which were visible on the graphical user interface when the user selected or deselected one or more outcome classification predictions to determine the one or more selected portions of the medical data. The method proceeds to step 312.

[0290] Step 312: Train PAM with recorded data

[0291] At step 312, the PAM 106P is updated utilising the selected and / or deselected outcome classification predictions and determined user parameters to generate an updated PAM. In other words, the PAM is trained based on the user selections in step 308 and determined user parameters in step 310.

[0292] Steps 300 to 312 may be repeated multiple times and / or for each abstracted case handled by the HDA.

[0293] Step 314: Dynamically adjust data

[0294] In step 314, the PAM 106P has been trained to identify and record selected elements of the abstracted case and correlate this with the declared or refuted outcome from the HDA in that case.

[0295] This enables the PAM 106P to prioritise certain data for the HDA which may lead to more efficient case processing. For example, over a number of training cycles, the PAM 106P can, in embodiments, be retrained to show the HDA specific parts of the EHR data for a given case in a particular order or configuration.

[0296] This may lead to the PAM 106P drawing attention to specific regions of the GUI or particular data elements of the abstraction case. This may be done by any suitable method, for example, by highlighting passages and places in the EHR data that supports the classification predictions of the FAM 106.

[0297] As such, each PAM 106P can highlight EHR text indicative of data of interest. For example, for matters such as a surgical site infection, the PAM 106P may draw the HDA’s attention to this part of the EHR chart text.Consider the example of the surgical site infection as described in step 302. For a further surgical site infection abstraction case, the PAM 106P may highlight data from the abstracted case such as:

[0298] 1 . A physician chart note entry stating that there is localized swelling (highlighting the specific part of the text describing this);

[0299] 2. A microbiology record indicating that a pathogenic bacteria was isolated from the wound culture;

[0300] The surgeon writes a chart entry stating that there is suspicion of a wound infection. This data may be prioritized in terms of display location, colour, spacing or other display configurations to draw the HDA’s attention thereto.

[0301] For example, particular data may be displayed in a particular order, location, highlighting or overlay to draw the HDA’s attention rapidly to relevant sections of the EHR data indicative of a given outcome. Consider the example above. If, in previous cases, the HDA has applied significant weighting to feature 2 (microbiology record) then data of this type may be displayed first or more prominently to encourage confident and accurate abstraction.

[0302] As a result, the dynamic attention adjustment thus enables the PAM 106P associated with a client device 60 of a particular HDA to learn the HDA’s particular or favourite workflow, and adapt the presentation of data for subsequent abstraction cases accordingly.

[0303] With each abstracted case completed by a HDA, the PAM 106P associated with the client device 60 of that HDA is operable to learn the workflows of the HDA and improve the processing of cases thereby. The method proceeds back to step 300 for a new case.

[0304] FURTHER METHOD EMBODIMENT

[0305] The following method relates to the steps occurring during operation of a surrogate benchmarking process using the medical analysis computing device 10. Figure 7 shows a flow chart of the method according to an embodiment.

[0306] Step 400: Obtain trained PAM 106P

[0307] At step 400, a trained PAM 106P is obtained. The PAM 106P may be trained through use as part of the abstraction process covered in steps 222 and / or steps 300 to 308 above.In other words, as each HDA processes EHR abstraction cases on their client device 60 alongside an associated PAM 106P, the associated PAM 106P is trained based on the responses of the HDA to each abstraction case. Over time, the local PAM 106P associated with a client device 60 of a particular HDA becomes a reflection of the HDA's own performance.

[0308] Step 402: Transfer local PAM

[0309] At step 402, the respective local PAM 106P is transferred to a centralised environment having a dataset with predetermined characteristics. In embodiments, the dataset may comprise verified and / or determined high accuracy “gold-standard” abstractions.

[0310] Step 404: Run local PAM on high accuracy dataset

[0311] At step 404, the PAM 106P is deployed on the gold-standard dataset. In other words, the PAM 106P is tasked with abstracting the verified dataset to generate a set of abstractions that can be compared in step 406.

[0312] Step 406: Compare PAM abstractions with “gold standard” dataset

[0313] At step 406, the PAM 106P abstractions are compared with the “gold-standard abstractions” to determine a variance therefrom. The variance of the individual PAM 106P from the gold standard then becomes an effective surrogate for the individual HDA’s abstraction performance.

[0314] Step 408: Generate performance metric

[0315] At step 408, a suitable metric of performance is generated based on the results of step 406. If the performance metric does not meet a predefined level or standard, then an alarm or notification may be generated. The alarm or notification may indicate that the HDA under assessment may require appropriate retraining.

[0316] Steps 400 to 408 may be repeated regularly to monitor HDA performance.Variations will be apparent to the skilled person. The methods described herein may be embodied in one or more pieces of software. The software is preferably held or otherwise encoded upon a memory device such as, but not limited to, any one or more of, a hard disk drive, RAM, ROM, solid state memory or other suitable memory device or component configured to software. The methods may be realised by executing / running the software. Additionally or alternatively, the methods may be hardware encoded.

[0317] The method encoded in software or hardware is preferably executed using one or more processors. The memory and / or hardware and / or processors are preferably comprised as at least part of one or more servers and / or other suitable computing systems.

[0318] In this specification, unless expressly otherwise indicated, the word "or" is used in the sense of an operator that returns a true value when either or both of the stated conditions are met, as opposed to the operator "exclusive or" which requires only that one of the conditions is met. The word "comprising" is used in the sense of "including" rather than to mean "consisting of".

[0319] All prior teachings above are hereby incorporated herein by reference. No acknowledgement of any prior published document herein should be taken to be an admission or representation that the teaching thereof was common general knowledge in Australia or elsewhere at the date thereof.

[0320] Where applicable, various embodiments provided by the present disclosure may be implemented using hardware, software, or combinations of hardware and software. Also, where applicable, the various hardware components and / or software components set forth herein may be combined into composite components comprising software, hardware, and / or both without departing from the spirit of the present disclosure. Where applicable, the various hardware components and / or software components set forth herein may be separated into sub-components comprising software, hardware, or both without departing from the scope of the present disclosure. In addition, where applicable, it is contemplated that software components may be implemented as hardware components and vice-versa.

[0321] Software, in accordance with the present disclosure, such as program code and / or data, may be stored on one or more computer readable mediums. It is also contemplated that software identified herein may be implemented using one or more general purpose or specific purpose computers and / or computer systems, networked and / or otherwise. Whereapplicable, the ordering of various steps described herein may be changed, combined into composite steps, and / or separated into sub-steps to provide features described herein.

[0322] While various operations have been described herein in terms of “modules,” “units” or “components,” it is noted that that terms are not limited to single units or functions. Moreover, functionality attributed to some of the modules or components described herein may be combined and attributed to fewer modules or components.

[0323] Further, whilst the present invention has been described with reference to specific embodiments and examples, those examples are intended to be illustrative only, and are not intended to limit the invention. It will be apparent to those of ordinary skill in the art that changes, additions or deletions may be made to the disclosed embodiments without departing from the scope of the invention.

Claims

CLAIMS1 . A computer-implemented method for determining an uncertainty metric for a medical data abstraction process utilising a federated machine learning model executed on a computing node and having a plurality of personal abstraction models (PAMs) associated therewith, wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and trained using a local training dataset stored on the respective client node, wherein the method is executed by at least one hardware processor and comprises the steps of:a) inputting medical data relating to a patient medical record into the federated machine learning model to generate an outcome classification prediction for one or more medical parameters;b) inputting the medical data relating to the patient medical record into each trained PAM to generate an outcome classification prediction for the one or more medical parameters for each trained PAM;c) utilising the determined outcome classification predictions from each trained PAM in an ensemble voting process to determine an uncertainty metric of the outcome classification prediction in step a);d) determining whether the uncertainty metric meets one or more predetermined criteria and, if so, accepting the determined outcome classification prediction and abstracting the patient medical record in an abstraction database.

2. A computer-implemented method according to claim 1 , wherein if the uncertainty metric does not meet the one or more predetermined criteria, the method further comprises: e) routing the patient medical record to a user for manual abstraction.

3. A computer-implemented method according to claim 1 or 2, wherein the ensemble voting process generates the uncertainty metric based on the relative agreement or disagreement of the outcome classification predictions of the trained PAMs.

4. A computer-implemented method according to any one of claims 1 to 3, wherein the uncertainty metric is further qualified utilising statistical analysis of the outcome classification prediction of the federated machine learning model.

5. A computer-implemented method according to claim 4, wherein the uncertainty metric is further qualified utilising statistical analysis of the outcome classification prediction of the federated machine learning model selected from the group of: confidence intervals;calculated model entropy; softmax probabilities in connection with margins of confidence; and Monte Carlo dropout determination.

6. A computer-implemented method according to any one of the preceding claims, wherein the federated machine learning model comprises a plurality of machine learning submodels.

7. A computer-implemented method according to claim 6, wherein the federated machine learning model comprises a plurality of chained machine learning sub-models for one or more medical parameters.

8. A computer-implemented method according to claim 6 or 7, wherein the federated machine learning model comprises a plurality of independent parallel sets of machine learning sub-models each configured to generate outcome classification predictions for specific medical parameters.

9. A computer-implemented method according to claim 8, wherein each set may comprise a single machine learning sub-model or a plurality of chained machine learning submodels.

10. A computer-implemented method according to any one of the preceding claims, wherein the one or more medical parameters comprise one or more of: a disease; an infection; a medical complication; a medical condition; a pharmaceutical complication; and a surgical complication.11 . A computer-implemented method according to any one of the preceding claims, wherein an outcome classification prediction comprises a probability that the outcome of the relevant medical parameter has occurred.

12. A computer-implemented method according to any one of the preceding claims, wherein the medical data comprises electronic health record (EHR) data.

13. A computer-implemented method according to claim 12, wherein the EHR data comprises one or more of: free medical text; medical notes; medical reports; chart notes; laboratory values; vital signs; diagnoses; and procedure codes.

14. A computer-implemented method according to any one of the preceding claims, wherein the federated machine learning model executed on the computing node comprises a global federated machine learning model.

15. A computer-implemented method according to claim 14, wherein the global federated machine learning model is a trained global federated machine learning model and wherein each PAM is further trained using a local training dataset stored on the respective client node, the local training dataset comprising abstraction outcomes selected by a user in relation to one or more medical parameters for a plurality of patient medical records.

16. A computer-implemented method according to any one of the preceding claims, wherein the computing node comprises a central computing node in a centralised federated learning environment.

17. A computer-implemented method according to any one of claims 1 to 15, wherein the computing node comprises a further client computing node in a decentralised federated learning environment.

18. A computer system comprising a computing node and a plurality of client computing nodes, the computing note being configured to execute a federated machine learning model thereon, the federated machine learning model being configured to perform data abstraction on input medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters and further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the computer system is operable to carry out the method of any one of claims 1 to 17.

19. A computer program comprising instructions which, when executed by a computer system, cause the computer to perform the steps of any one of claims 1 to 17.

20. A computer-implemented method of medical data abstraction performance benchmarking utilising a federated machine learning model executed on a computing node and configured to perform data abstraction on input medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters, the federated machine learning model further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the method is executed by at least one hardware processor and comprises the steps of:a) training a PAM on a client computing node using a local training dataset to generate a trained PAM, the local training dataset comprising abstraction outcomes selected by a user in relation to one or more medical parameters for a plurality of patient medical records; b) transferring the trained PAM to a benchmarking computing environment;c) performing, using the trained PAM, data abstraction on a benchmark input medical dataset to generate an outcome classification prediction in relation to one or more medical parameters;d) comparing the outcome classification prediction from the trained PAM with a true outcome classification prediction to determine a performance metric for the user associated with the trained PAM.

21. A computer-implemented method according to claim 20, wherein the federated machine learning model comprises a plurality of machine learning sub-models.

22. A computer-implemented method according to claim 21 , wherein the federated machine learning model comprises a plurality of chained machine learning sub-models for one or more medical parameters.

23. A computer-implemented method according to claim 21 or 22, wherein the federated machine learning model comprises a plurality of independent parallel sets of machine learning sub-models each configured to generate outcome classification predictions for specific medical parameters.

24. A computer-implemented method according to claim 23, wherein each set may comprise a single machine learning sub-model or a plurality of chained machine learning sub-models.

25. A computer-implemented method according to claim 23 or 24, wherein when the federated machine learning model is configured to generate outcome classification predictions for one or more specific medical parameters associated with a particular set of sub-model(s), the remaining sets are not activated.

26. A computer-implemented method according to any one of claims 20 to 25, wherein the performance metric comprises an accuracy metric or error rate metric.

27. A computer-implemented method according to any one of claims 20 to 26, wherein the one or more medical parameters comprise one or more of: a disease; an infection; amedical complication; a medical condition; a pharmaceutical complication; and a surgical complication.

28. A computer-implemented method according to any one of claims 20 to 27, wherein an outcome classification prediction comprises a probability that the outcome of the relevant medical parameter has occurred.

29. A computer-implemented method according to any one of claims 20 to 28, wherein the medical data comprises electronic health record (EHR) data.

30. A computer-implemented method according to claim 29, wherein the EHR data comprises one or more of: free medical text; medical notes; medical reports; chart notes; laboratory values; vital signs; diagnoses; and procedure codes.31 . A computer-implemented method according to any one of claims 20 to 30, wherein the federated machine learning model executed on the computing node comprises a global federated machine learning model.

32. A computer-implemented method according to any one of claims 20 to 31 , wherein the benchmarking computing environment is executed on the computing node.

33. A computer-implemented method according to any one of claims 20 to 32, wherein the computing node comprises a central computing node in a centralised federated learning environment.

34. A computer-implemented method according to any one of claims 20 to 32, wherein the computing node comprises a further client computing node in a decentralised federated learning environment.

35. A computer system comprising a computing node and a plurality of client computing nodes, the computing note being configured to execute a federated machine learning model thereon, the federated machine learning model being configured to perform data abstraction on input medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters and further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the computer system is operable to carry out the method of any one of claims 20 to 34.

36. A computer program comprising instructions which, when executed by a computer system, cause the computer to perform the steps of any one of claims 20 to 34.

37. A computer-implemented method of training a federated machine learning model executed on a computing node and configured to perform data abstraction on input medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters, the federated machine learning model further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the method is executed by at least one hardware processor and comprises the steps of:a) receiving, at a client computing node, medical data relating to a patient medical record;b) inputting the medical data relating to the patient medical record into the PAM associated with the client computing node to generate one or more outcome classification predictions for one or more medical parameters;c) displaying, on a graphical user interface associated with the client computing node, the outcome classification predictions for the one or more medical parameters and at least a part of the medical data associated with the patient medical record;d) receiving one or more user input actions to select and / or deselect the one or more outcome classification predictions for the one or more medical parameters;e) determining user parameter data representative of the user’s interaction with or analysis of the medical data of the patient medical record;f) updating the PAM utilising the selected and / or deselected outcome classification predictions and determined user parameters to generate an updated PAM.

38. A computer-implemented method according to claim 37, wherein an abstraction session for abstraction of the patient medical record is initiated at step c) and terminated following selection and / or deselection of all available outcome classification predictions for one or more medical parameters in step d).

39. A computer-implemented method according to claim 38, wherein the user parameter data comprises one or more selected portions of the medical data of the patient medical record that the user has been determined to have focussed on during the abstraction session.

40. A computer-implemented method according to claim 39, wherein step e) comprises determining the time period for which different portions of the medical data of the patient medical record are visible on the graphical user interface during the abstraction session.41 . A computer-implemented method according to claim 40, wherein one or more portions of the medical data of the patient medical record which were visible on the graphical user interface for longest time during the abstraction session are selected as the one or more selected portions of the medical data.

42. A computer-implemented method according to claim 41 , wherein step e) comprises determining the portion of the medical data of the patient medical record are visible on the graphical user interface when the user selected or deselected one or more outcome classification predictions to determine the one or more selected portions of the medical data.

43. A computer-implemented method according to any one of claims 37 to 42, wherein, for any selected outcome classification predictions in step d), the method further comprises: g) generating an abstracted medical record outcome and storing the abstracted medical record outcome in an abstraction database.

44. A computer-implemented method according to any one of claims 37 to 43, wherein the method further comprises:h) identifying, on the graphical user interface, one or more elements of the medical data associated with the determination of the outcome classification predictions for the one or more medical parameters;45. A computer-implemented method according to claim 44, wherein the step of identifying one or more elements of the medical data in step h) comprises highlighting the one or more elements of the medical data on the graphical user interface.

46. A computer-implemented method according to claim 45, wherein the highlighting comprises displaying the one or more elements in a different colour, font, location and / or format from the elements of the medical data having no or a lesser association with the determination of the outcome classification predictions for the one or more medical parameters.

47. A computer-implemented method according to any one of claims 37 to 46, wherein steps a) to f) are repeated for subsequent patient medical records.

48. A computer-implemented method according to any one of claims 37 to 47, wherein the federated machine learning model comprises a plurality of machine learning sub-models.

49. A computer-implemented method according to claim 48, wherein the federated machine learning model comprises a plurality of chained machine learning sub-models for one or more medical parameters.

50. A computer-implemented method according to claim 48 or 49, wherein the federated machine learning model comprises a plurality of independent parallel sets of machine learning sub-models each configured to generate outcome classification predictions for specific medical parameters.

51. A computer-implemented method according to claim 50, wherein each set may comprise a single machine learning sub-model or a plurality of chained machine learning sub-models.

52. A computer-implemented method according to claim 50 or 51 , wherein when the federated machine learning model is configured to generate outcome classification predictions for one or more specific medical parameters associated with a particular set of sub-model(s), the remaining sets are not activated.

53. A computer-implemented method according to any one of claims 37 to 52, wherein the one or more medical parameters comprise one or more of: a disease; an infection; a medical complication; a medical condition; a pharmaceutical complication; and a surgical complication.

54. A computer-implemented method according to any one of claims 37 to 53, wherein an outcome classification prediction comprises a probability that the outcome of the relevant medical parameter has occurred.

55. A computer-implemented method according to any one of claims 37 to 54, wherein the medical data comprises electronic health record (EHR) data.

56. A computer-implemented method according to claim 55, wherein the EHR data comprises one or more of: free medical text; medical notes; medical reports; chart notes; laboratory values; vital signs; diagnoses; and procedure codes.

57. A computer-implemented method according to any one of claims 37 to 56, wherein the federated machine learning model executed on the computing node comprises a global federated machine learning model.

58. A computer-implemented method according to claim 56, wherein the global federated machine learning model is a trained global federated machine learning model and whereineach PAM is configured to be further trained using the local training dataset stored on the respective client node.

59. A computer-implemented method according to any one of claims 37 to 58, wherein the computing node comprises a central computing node in a centralised federated learning environment.

60. A computer-implemented method according to any one of claims 37 to 58, wherein the computing node comprises a further client computing node in a decentralised federated learning environment.61 . A computer system comprising a computing node and a plurality of client computing nodes, the computing note being configured to execute a federated machine learning model thereon, the federated machine learning model being configured to perform data abstraction on input medical data relating to one or more patient medical records to generate an outcome classification prediction for one or more medical parameters and further comprising a plurality of personal abstraction models (PAMs), wherein each PAM comprises an instance of the federated machine learning model local to and executed on a client computing node and configured to be trained using a local training dataset stored on the respective client node, wherein the computer system is operable to carry out the method of any one of claims 37 to 60.

62. A computer program comprising instructions which, when executed by a computer system, cause the computer to perform the steps of any one of claims 37 to 60.

Images