Clinical deterioration event prediction

Abstract

A method of predicting a clinical deterioration event for a subject, comprising: obtaining time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point; generating an inference feature dataset including generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject; and predicting, using a deterioration prediction machine learning model, a clinical deterioration event for the subject based on the inference feature dataset. There is also provided a corresponding method of training a deterioration prediction machine learning model, as well as a corresponding system for predicting a clinical deterioration event.

Description

CLINICAL DETERIORATION EVENT PREDICTIONCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority of Singapore Patent Application No.10202500367T filed on 10 February 2025, the content of which being hereby incorporated by reference in its entirety for all purposes.TECHNICAL FIELD

[0002] The present invention generally relates to clinical deterioration event prediction, and more particularly, a method and a system for predicting a clinical deterioration event for a subject, and a method and a system for training a deterioration prediction machine learning model.BACKGROUND

[0003] Clinical deterioration refers to the worsening of a patient’s physiological condition, often characterized by significant deviations from baseline health status. This clinical deterioration process is marked by declining vital signs, such as unstable blood pressure, elevated heart rate, respiratory distress, and / or low oxygen saturation. Left unrecognized or unmanaged, clinical deterioration can lead to severe consequences, including organ failure, cardiac arrest, or even death. In hospital settings, it remains a major cause of preventable mortality, especially in general care units where patients may not be under continuous monitoring like in intensive care units (ICUs). The timely identification / detection of clinical deterioration is critical, as delayed recognition and intervention are associated with poor patient outcomes and increased healthcare costs.

[0004] Vital signs serve as key indicators of a patient’s health status and are often the first signs of clinical deterioration. Changes such as a rapid heart rate (tachycardia), irregular breathing patterns, or a drop in oxygen saturation levels can indicate the onset of critical conditions. These physiological changes typically occur hours before severe events, such as cardiac arrest or the need for mechanical ventilation, thereby providing an opportunity for early detection. Research shows that early intervention based on these warning signs, such as activating a Rapid Response Team (RRT) or transferring patients to a higher level of care, can significantly reduce mortality rates and improve patient outcomes. In particular, in the field of healthcare, early identification of patient deterioration is crucial for timely intervention andimproved patient outcomes. Prompt recognition and timely action not only save lives but also prevent complications, shorten hospital stays, and enhance overall quality of care.

[0005] Traditional monitoring systems often rely on static thresholds and single-point measurements, which may not capture the dynamic nature of patient conditions. Machine learning methods offer a promising approach to enhance the accuracy and timeliness of detecting patient deterioration. However, existing machine learning methods often struggle with the complexity and variability of physiological data. For example, numerous early warning systems, including tools like the Modified Early Warning Score (MEWS) and the National Early Warning Score (NEWS), have been developed to predict clinical deterioration. While these systems have improved the recognition of at-risk patients, many suffer from a low positive predictive value (PPV), leading to frequent false alarms. A low PPV will contribute to alarm fatigue among healthcare providers, with excessive alerts desensitizing staff and delaying responses to true emergencies.

[0006] A need therefore exists to provide a method and a system for predicting a clinical deterioration event for a subject, as well as a method and a system of training a deterioration prediction machine learning model, that seek to overcome, or at least ameliorate, one or more deficiencies in existing methods or systems, and more particularly, with improved prediction accuracy. It is against this background that the present invention has been developed.SUMMARY

[0007] According to a first aspect of the present invention, there is provided a method of predicting a clinical deterioration event for a subject using at least one processor, the method comprising:obtaining time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point, wherein the end time point is a prediction time point at which the clinical deterioration event for the subject is being predicted;generating an inference feature dataset comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject, wherein the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window to the end time point has a longer time period than the time periodof the first time window, and the inference feature dataset comprises the first and second sets of vital sign statistical features; andpredicting, using a deterioration prediction machine learning model, a clinical deterioration event for the subject based on the inference feature dataset.

[0008] According to a second aspect of the present invention, there is provided a method of training a deterioration prediction machine learning model using at least one processor, the method comprising:obtaining time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point;generating a training feature dataset comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject, wherein the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window to the end time point has a longer time period than the time period of the first time window and the training feature dataset comprises the first and second sets of vital sign statistical features;labelling the training feature dataset as a positive class or a negative class based on whether the time-series data of vital sign measurements is associated with a clinical deterioration event or not; andtraining the deterioration prediction machine learning model based on the labelled training feature dataset for predicting a clinical deterioration event for a subject.

[0009] According to a third aspect of the present invention, there is provided a system for predicting a clinical deterioration event for a subject, the system comprising:at least one memory; andat least one processor communicatively coupled to the at least one memory and configured to perform the method of predicting a clinical deterioration event for a subject according to the above-mentioned first aspect of the present invention.

[0010] According to a fourth aspect of the present invention, there is provided a system for training a deterioration prediction machine learning model, the system comprising:at least one memory; andat least one processor communicatively coupled to the at least one memory and configured to perform the method of training a deterioration prediction machine learning model according to the above-mentioned second aspect of the present invention.

[0011] According to a fifth aspect of the present invention, there is provided a computer program product, embodied in one or more non-transitory computer-readable storage mediums, comprising instructions executable by at least one processor to perform the method of predicting a clinical deterioration event for a subject according to the above-mentioned first aspect of the present invention.

[0012] According to a sixth aspect of the present invention, there is provided a computer program product, embodied in one or more non-transitory computer-readable storage mediums, comprising instructions executable by at least one processor to perform the method of training a deterioration prediction machine learning model according to the above-mentioned second aspect of the present invention.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments of the present invention will be better understood and readily apparent to one of ordinary skill in the art from the following written description, by way of example only, and in conjunction with the drawings, in which:FIG. 1 depicts a schematic diagram of a method of predicting a clinical deterioration event for a subject, according to various embodiments of the present invention;FIG. 2 depicts a schematic diagram of a method of training a deterioration prediction machine learning model, according to various embodiments of the present invention;FIG. 3 depicts a schematic block diagram of a system for predicting a clinical deterioration event for a subject, according to various embodiments of the present invention;FIG. 4 depicts a schematic block diagram of a system for training a deterioration prediction machine learning model, according to various embodiments of the present invention;FIG. 5 depicts a schematic drawing of an example method of predicting a clinical deterioration event for a subject, according to various example embodiments of the present invention;FIG. 6A shows a schematic drawing of time-series data of vital sign measurements of a subject obtained, with an example first time window and an example second time window for illustrating a dual-window feature extraction process, according to various example embodiments of the present invention;FIG. 6B shows a schematic drawing of time-series data of vital sign measurements of a subject obtained, with example four time windows for illustrating a four-window feature extraction process, according to various example embodiments of the present invention;FIG. 6C shows a schematic drawing of the time-series data of vital sign measurements shown in FIG. 6 A but with a different set of vital sign statistical features (i.e., based on a different combination of types (including a different number of types) of vital sign measurements) extracted from each time window, according to various example embodiments of the present invention;FIG. 6D shows a schematic drawing of time-series data of vital sign measurements of a subject, with an example first / current time window shifting and an example second / baseline time window growing over time;FIG. 7 shows an example inference feature dataset generated, according to various example embodiments of the present invention;FIG. 8 shows a graphical representation of ROC curve, PR curve and normalized alert rate of the four-vitals model according to various example embodiments of the present invention and Deterioration Index (DI);FIGs. 9A and 9B show the ROC curves, PR curves and normalized alert rate of the four-vital sign model stratified by hospital site (FIG. 9A) and race (FIG. 9B), according to various example embodiments of the present invention;FIG. 10 shows the ROC curve, PR curve and normalized alert rate of the one-vital, two-vitals, three-vitals and four-vitals models, according to various example embodiments of the present invention; andFIG. 11 shows descriptive statistics of the validation set and the train set stratified by hospital.DETAILED DESCRIPTION

[0014] Various embodiments of the present invention provide clinical deterioration event prediction, and more particularly, a method and a system for predicting a clinical deterioration event for a subject (e.g., a patient), and a method and a system for training a deterioration prediction machine learning model.

[0015] As discussed in the background, while early warning systems have been developed to predict clinical deterioration, they suffer from a low positive predictive value (PPV), leading to frequent false alarms. A low PPV will contribute to alarm fatigue among healthcareproviders, with excessive alerts desensitizing staff and delaying responses to true emergencies. In this regard, various example embodiments provide a method and a system for predicting a clinical deterioration event for a subject, as well as a method and a system of training a deterioration prediction machine learning model, that seek to overcome, or at least ameliorate, one or more deficiencies in existing methods or systems, and more particularly, with improved prediction accuracy.

[0016] FIG. 1 depicts a schematic diagram of a method 100 of predicting a clinical deterioration event for a subject (e.g., a patient) using at least one processor, according to various embodiments of the present invention. The method 100 comprises obtaining (at 106) time-series data of vital sign measurements of a subject with respect to a time interval defined (or bounded) by a start time point and an end time point. The end time point is a prediction time point (or inference time point) at which the clinical deterioration event for the subject is being predicted. The method 100 further comprises generating (at 108) an inference feature dataset comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject. In particular, the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window (i.e., the end opposite to a start of the second time window) to the end time point has a longer time period than the time period of the first time window. In this regard, the inference feature dataset comprises the first and second sets of vital sign statistical features. The method 100 further comprises predicting (at 110), using a deterioration prediction machine learning model, a clinical deterioration event for the subject based on the inference feature dataset.

[0017] It will be appreciated by a person skilled in the art that the clinical deterioration event for a subject may be defined as desired or as appropriate, as long as it is an event relating to a clinical deterioration of a subject (worsening of a subject’s physiological condition) that is desired to be predicted. By way of examples only and without limitation, the clinical deterioration event for a subject may be a certain degree or level of worsening of the subject’s physiological condition, a Rapid Response Team (RRT) call for the subject or an unplanned intensive care unit (ICU) transfer of the subject. Accordingly, it will be appreciated by a person skilled in the art that the clinical deterioration event is not limited to any particular or specific type of clinical deterioration event and may be defined as desired or as appropriate. Furthermore, it will be appreciated by a person skilled in the art that the method 100 predictsthe occurrence of a clinical deterioration event for a subject in the future, and the time period in the future may be defined as desired or as appropriate (e.g., in the next 30 minutes, 1 hour, 2 hours, 6 hours, 12 hours, or 24 hours, etc.).

[0018] The method 100 of predicting a clinical deterioration event has improved prediction accuracy. In particular, the method 100 generates (or extracts) vital sign statistical features from at least two time windows of the time-series data of vital sign measurements covering different time segments or zones of the above-mentioned time interval. In particular, because the above-mentioned first time period is configured to have a longer time period (e.g., 4 to 48 times larger) than the time period of the first time window, the vital sign statistical features are generated from the first time window (having a time period that ends at the end time point of the time interval) based on more recent vital sign measurements (e.g., resulting in shorter-term vital sign trend features) compared to the vital sign statistical features generated from the second time window (having a time period that starts at the start time point of the time interval), which are based on older vital sign measurements (obtained longer ago) (e.g., resulting in longer-term vital sign trend features). This enables the method 100 to generate (or extract) vital sign statistical features over different time segments or zones, which has been surprisingly found to improve (e.g., significantly improve) accuracy in predicting, using a deterioration prediction machine learning model, a clinical deterioration event for the subject based on such vital sign statistical features generated. Furthermore, since the above-mentioned first time period is configured to have a set (predefined) time period longer than the time period of the first time window, the time period or length of the second time window grows over time as the timeseries data of vital sign measurements (or the number of vital sign measurements) grows over time, thereby enabling a growing number of past vital sign measurements (starting from the start time point of the time interval) to be captured or utilized to, for example, establish better baseline or longer-term trends (vital sign trend features). This has been surprisingly found to improve (e.g., significantly improve) accuracy in predicting, using the deterioration prediction machine learning model, a clinical deterioration event for the subject. These advantages or technical effects, and / or other advantages or technical effects, will become more apparent to a person skilled in the art as the method 100 of (and the corresponding system for) predicting a clinical deterioration event, as well as the corresponding method of (and the corresponding system for) training a deterioration prediction machine learning model, is described in more detail according to various embodiments and example embodiments of the present invention.

[0019] In various embodiments, the first time period is about 4 to 48 times larger than the time period of the first time window. In this regard, the first time period may be set according to a desired or an optimized time separation or gap between the first and second time windows, such as based on a desired or an optimized time difference between recent vital sign measurements (e.g., for generating shorter-term vital sign trend features) and older vital sign measurements (e.g., for generating longer-term vital sign trend features), for example, for optimizing trend differentiation. In various embodiments, the first time period may be about 6 to 46 times larger, about 8 to 44 times larger, about 10 to 42 times larger, about 12 to 40 times larger, about 14 to 38 times larger, about 16 to 36 times larger, about 18 to 34 times larger, about 20 to 32 times larger, about 22 to 30 times larger, about 22 to 28 times larger, about 22 to 26 times larger or about 23 to 25 times larger than the time period of the first time window.

[0020] In various embodiments, the time period of the first time window is in a range of about 30 minutes to about 24 hours, and the first time period is in a range of about 4 hours to about 48 hours. In various embodiments, the time period of the first time window may be in a range of about 30 minutes to about 20 hours, about 30 minutes to about 18 hours, about 35 minutes to about 16 hours, about 35 minutes to about 14 hours, about 40 minutes to about 12 hours, about 40 minutes to about 10 hours, about 40 minutes to about 8 hours, about 45 minutes to about 6 hours, about 45 minutes to about 4 hours, about 45 minutes to about 2 hours, about 45 minutes to about 90 minutes or about 50 minutes to about 70 minutes, and the first time period may be in a range of about 6 hours to about 46 hours, about 8 hours to about 44 hours, about 10 hours to about 42 hours, about 12 hours to about 40 hours, about 14 hours to about 38 hours, about 14 hours to about 38 hours, about 16 hours to about 36 hours, about 18 hours to about 34 hours, about 20 hours to about 32 hours, about 22 hours to about 30 hours, about 22 hours to about 28 hours, about 22 hours to about 26 hours or about 23 hours to about 25 hours. In various embodiments, the time period of the first time window is about 1 hour, and the first time period is about 24 hours. In this regard, they are determined to be optimal configurations according to various embodiments of the present invention.

[0021] In various embodiments, the first time window comprises one or more measurement time points (at which vital sign measurements of the subject are taken), and the second time window comprises a plurality of measurement time points (at which vital sign measurements of the subject are taken).

[0022] In various embodiments, the above-mentioned generating (at 108) the first set of vital sign statistical features from the first time window comprises determining the first set ofvital sign statistical features based on vital sign measurements at the one or more measurement time points of the first time window. Similarly, the above-mentioned generating (at 108) the second set of vital sign statistical features from the second time window comprises determining the second set of vital sign statistical features based on vital sign measurements at the plurality of measurement time points of the second time window.

[0023] In various embodiments, the first set of vital sign statistical features is a first set of vital sign trend features, and the second set of vital sign statistical features is a second set of vital sign trend features. Accordingly, in various embodiments, the method 100 advantageously generates (or extracts) vital sign trend features over different time segments or zones of the time interval (e.g., resulting in shorter-term vital sign trend features and longer-term vital sign trend features), which has been surprisingly found to improve (e.g., significantly improve) accuracy in predicting, using a deterioration prediction machine learning model, a clinical deterioration event for the subject based on such vital sign trend features generated.

[0024] In various embodiments, the vital sign measurements (at each measurement time point) comprise at least two different types of vital sign measurements of the subject. In this regard, for each type of vital sign measurements, each of the first and second sets of vital sign statistical features comprises a plurality of different types of vital sign statistical features derived from the type of vital sign measurements. In various embodiments, the vital sign measurements comprise at least three different types of vital sign measurements of the subject. In various embodiments, the vital sign measurements comprise at least four different types of vital sign measurements of the subject.

[0025] In various embodiments, the vital sign measurements comprise two or more of respiration rate measurements, heart rate measurements, oxygen saturation measurements, blood pressure measurements and temperature measurements of the subject.

[0026] In various embodiments, each of the plurality of different types of vital sign statistical features derived from the type of vital sign measurements is generated based on a mean, a standard deviation, a minimum, a maximum, a median, a quartile (e.g., 1stquartile (QI) or 3rdquartile (Q3)), an interquartile range, a mean absolute deviation, a variance or a mode (value that appears most often / frequently) of the type of vital sign measurements.

[0027] In various embodiments, the above-mentioned generating (at 108) the inference dataset further comprises extracting first measurement interval information from the first time window and second measurement interval information from the second time window. In this regard, the inference feature dataset further comprises the first and second measurement intervalinformation. In various embodiments, the measurement interval information extracted from a time window comprises, for each pair of consecutive measurement time points in the time window, the time interval length between the pair of consecutive measurement time points. In this regard, predicting the clinical deterioration event for the subject further based on the first and second measurement interval information has been found to further improve prediction accuracy.

[0028] In various embodiments, the inference feature dataset further comprises the vital sign measurements (i.e., raw vital sign measurements) at a measurement time point at or nearest to the prediction time point. In this regard, predicting the clinical deterioration event for the subject further based on the raw vital sign measurements at a measurement time point at or nearest to the prediction time point has been found to further improve prediction accuracy.

[0029] In various embodiments, the above-mentioned predicting (at 110) the clinical deterioration event comprises estimating, using the deterioration prediction machine learning model, a probability of the clinical deterioration event for the subject based on the inference feature dataset.

[0030] In various embodiments, the above-mentioned generating (at 108) the inference feature dataset further comprises generating one or more additional sets of vital sign statistical features respectively from one or more additional time windows of the time-series data of the vital sign measurements of the subject. The first time window, the second time window, and the one or more additional time windows have different time periods. In this regard, the inference feature dataset further comprises the one or more additional sets of vital sign statistical features. In various embodiments, each additional set of vital sign statistical features may be generated in the same or similar manner as the first or second set of vital sign statistical features as described above according to various embodiments of the present invention, and thus need not be repeated for clarity and conciseness. In various embodiments, the one or more additional time windows comprise one or more pairs of additional time windows, each pair of additional time windows comprises a first additional time window and a second additional time window wherein the first additional time window has a time period that ends at the end time point, the second additional time window has a time period that starts at the start time point, and a second time period from an end of the second additional time window (i.e., the end opposite to a start of the second additional time window) to the end time point has a longer time period than the time period of the first additional time window. Similar to the above-mentioned the first time period, the second time period may also be set according to a desired or an optimized timeseparation or gap between the first and second additional time windows, such as based on a desired or an optimized time difference between recent vital sign measurements (e.g., for generating shorter-term vital sign trend features) and older vital sign measurements (e.g., for generating longer-term vital sign trend features), for example, for optimizing trend differentiation. In this regard, the first and second time periods may be optimized together.

[0031] In various embodiments, the method 100 is performed iteratively over time at each new prediction time point at which the clinical deterioration event for the subject is being predicted. In this regard, for each iteration for a new prediction time point, the time-series data of vital sign measurements of the subject is obtained with respect to a time interval defined (or bounded) by the start time point and the end time point, the end time point being the new prediction time point. In various embodiments, the above-mentioned each new prediction time point corresponds to each new measurement time point. That is, the method 100 may be configured to predict a clinical deterioration event for the subject over time (e.g., until a clinical deterioration event is predicted or occurs) at each new measurement time point (i.e., each time vital sign measurements are obtained from the subject). Therefore, the method 100 may advantageously be employed to continuously or repeatedly monitor the subject for clinical deterioration event.

[0032] FIG. 2 depicts a schematic diagram of a method 200 of training a deterioration prediction machine learning model using at least one processor, according to various embodiments of the present invention. In particular, the method 200 trains the deterioration prediction machine learning model for use by the method 100 to predict the clinical deterioration event for a subject (e.g., a patient). The method 200 comprises: obtaining (at 206) time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point; and generating (at 208) a training feature dataset comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject. In particular, the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window (i.e., the end opposite to a start of the second time window) to the end time point has a longer time period than the time period of the first time window. In this regard, the training feature dataset comprises the first and second sets of vital sign statistical features. The method 200 further comprises labelling (at 210) the training feature dataset as a positive class or a negative class based on whether thetime-series data of vital sign measurements is associated with a clinical deterioration event or not; and training (at 212) the deterioration prediction machine learning model based on the labelled training feature dataset for predicting a clinical deterioration event for a subject.

[0033] As explained hereinbefore, it will be appreciated by a person skilled in the art that the clinical deterioration event for a subject may be defined as desired or as appropriate, as long as it is an event relating to a clinical deterioration of a subject (worsening of a subject’s physiological condition) that is desired to be predicted. Accordingly, it will be appreciated by a person skilled in the art that the clinical deterioration event is not limited to any particular or specific type of clinical deterioration event and may be defined as desired or as appropriate. Furthermore, it will be appreciated by a person skilled in the art that the method 200 trains the deterioration prediction machine learning model for predicting the occurrence of a clinical deterioration event for a subject (e.g., a patient) in the future, and the time period in the future may be defined as desired or as appropriate (e.g., in the next 30 minutes, 1 hour, 2 hours, 6 hours, 12 hours, or 24 hours, etc). For example, the training feature dataset may be labelled as a positive class or a negative class based on whether the time-series data of vital sign measurements is associated with a clinical deterioration event as defined or not. In various embodiments, the method 200 of training the deterioration prediction machine learning model may be performed for each time-series data of vital sign measurements obtained.

[0034] In the same or similar manner as described hereinbefore for the method 100 of predicting a clinical deterioration event according to various embodiments of the present invention, the method 200 of training a deterioration prediction machine learning model produces a trained deterioration prediction machine learning model having improved prediction accuracy.

[0035] In various embodiments, similar to the method 100, the first time period is about 4 to 48 times larger than the time period of the first time window.

[0036] In various embodiments, similar to the method 100, the time period of the first time window is in a range of about 30 minutes to about 24 hours, and the first time period is in a range of about 4 hours to about 48 hours.

[0037] In various embodiments, similar to the method 100, the time period of the first time window is about 1 hour, and the first time period is about 24 hours.

[0038] In various embodiments, similar to the method 100, the first time window comprises one or more measurement time points (at which vital sign measurements of the subject aretaken), and the second time window comprises a plurality of measurement time points (at which vital sign measurements of the subject are taken).

[0039] In various embodiments, similar to the method 100, the above-mentioned generating (at 208) the first set of vital sign statistical features from the first time window comprises determining the first set of vital sign statistical features based on vital sign measurements at the one or more measurement time points of the first time window. Similarly, the above-mentioned generating (at 208) the second set of vital sign statistical features from the second time window comprises determining the second set of vital sign statistical features based on vital sign measurements at the plurality of measurement time points of the second time window.

[0040] In various embodiments, similar to the method 100, the first set of vital sign statistical features is a first set of vital sign trend features, and the second set of vital sign statistical features is a second set of vital sign trend features.

[0041] In various embodiments, similar to the method 100, the vital sign measurements (at each measurement time point) comprise at least two different types of vital sign measurements of the subject. In this regard, for each type of vital sign measurements, each of the first and second sets of vital sign statistical features comprises a plurality of different types of vital sign statistical features derived from the type of vital sign measurements. In various embodiments, similar to the method 100, the vital sign measurements comprise at least four different types of vital sign measurements of the subject.

[0042] In various embodiments, similar to the method 100, the vital sign measurements comprise two or more of respiration rate measurements, heart rate measurements, oxygen saturation measurements, blood pressure measurements and temperature measurements of the subject.

[0043] In various embodiments, similar to the method 100, each of the plurality of different types of vital sign statistical features derived from the type of vital sign measurements is generated based on a mean, a standard deviation, a minimum, a maximum, a median, a quartile, an interquartile range, a mean absolute deviation, a variance or a mode (value that appears most often / frequently) of the type of vital sign measurements.

[0044] In various embodiments, similar to the method 100, the above-mentioned generating (at 212) the training feature dataset further comprises extracting first measurement interval information from the first time window and second measurement interval information from the second time window. In this regard, the training feature dataset further comprises the first and second measurement interval information. In various embodiments, the measurement intervalinformation extracted from a time window comprises, for each pair of consecutive measurement time points in the time window, the time interval length between the pair of consecutive measurement time points.

[0045] In various embodiments, similar to the method 100, the training feature dataset further comprises the vital sign measurements (i.e., raw vital sign measurements) at a measurement time point at or nearest to the end time point.

[0046] In various embodiments, similar to the method 100, the above-mentioned predicting the clinical deterioration event for the subject comprises estimating a probability of the clinical deterioration event for the subject.

[0047] In various embodiments, similar to the method 100, the above-mentioned generating (at 208) the training feature dataset further comprises generating one or more additional sets of vital sign statistical features respectively from one or more additional time windows of the timeseries data of the vital sign measurements of the subject. The first time window, the second time window, and the one or more additional time windows have different time periods. In this regard, the training feature dataset further comprises the one or more additional sets of vital sign statistical features. In various embodiments, each additional set of vital sign statistical features may be generated in the same or similar manner as the first or second set of vital sign statistical features as described above according to various embodiments of the present invention, and thus need not be repeated for clarity and conciseness. In various embodiments, the one or more additional time windows comprise one or more pairs of additional time windows, each pair of additional time windows comprises a first additional time window and a second additional time window wherein the first additional time window has a time period that ends at the end time point, the second additional time window has a time period that starts at the start time point, and a second time period from an end of the second additional time window (i.e., the end opposite to a start of the second additional time window) to the end time point has a longer time period than the time period of the first additional time window.

[0048] In various embodiments, the time-series data of vital sign measurements is associated with a clinical deterioration event and the end time point corresponds to a start (or an onset) of the clinical deterioration event. In various embodiments, the time-series data of vital sign measurements is not associated with a clinical deterioration event and the end time point is a time point randomly selected in the time interval.

[0049] FIG. 3 depicts a schematic block diagram of a system 300 for predicting a clinical deterioration event for a subject, according to various embodiments of the present invention,corresponding to the above-mentioned method 100 of predicting a clinical deterioration event for a subject as described hereinbefore with reference to FIG. 1 according to various embodiments of the present invention. The system 300 comprises: at least one memory 302; and at least one processor 304 communicatively coupled to the at least one memory 302 and configured to perform the method 100 of predicting a clinical deterioration event for a subject according to various embodiments of the present invention. Accordingly, the at least one processor 304 is configured to: obtain time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point, wherein the end time point is a prediction time point at which the clinical deterioration event for the subject is being predicted; generate an inference feature dataset comprising generate a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject, wherein the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window to the end time point has a longer time period than the time period of the first time window, and the inference feature dataset comprises the first and second sets of vital sign statistical features; and predict, using a deterioration prediction machine learning model, a clinical deterioration event for the subject based on the inference feature dataset.

[0050] It will be appreciated by a person skilled in the art that the at least one processor 304 may be configured to perform various functions or operations through set(s) of instructions (e.g., software modules) executable by the at least one processor 304 to perform various functions or operations. Accordingly, as shown in FIG. 3, the system 300 may comprise: a vital sign measurement obtaining module (or a vital sign measurement obtaining circuit) 306 configured to obtain time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point, wherein the end time point is a prediction time point at which the clinical deterioration event for the subject is being predicted; an inference feature dataset generating module (or an inference feature dataset generating circuit) 308 configured to generate an inference feature dataset comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject, wherein the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window to the end time point has a longertime period than the time period of the first time window, and the inference feature dataset comprises the first and second sets of vital sign statistical features; and a deterioration prediction module (or a deterioration prediction circuit) 310 configured to predict, using a deterioration prediction machine learning model, a clinical deterioration event for the subject based on the inference feature dataset.

[0051] It will be appreciated by a person skilled in the art that the above-mentioned modules of the system 300 are not necessarily separate modules, and two or more modules may be realized by or implemented as one functional module (e.g., a circuit or a software program) as desired or as appropriate without deviating from the scope of the present invention. For example, two or more of the vital sign measurement obtaining module 306, the inference feature dataset generating module 308 and the deterioration prediction module 310 may be realized (e.g., compiled together) as one executable software program (e.g., software application), which for example may be stored in the at least one memory 302 and executable by the at least one processor 304 to perform the corresponding functions or operations as described herein according to various embodiments of the present invention.

[0052] In various embodiments, the system 300 for predicting a clinical deterioration event corresponds to the method 100 of predicting a clinical deterioration event as described hereinbefore with reference to FIG. 1, therefore, various operations, functions or steps configured to be performed by the at least one processor 304 may correspond to various operations, functions or steps of the method 100 of predicting a clinical deterioration event described hereinbefore according to various embodiments, and thus need not be repeated with respect to the system 300 for predicting a clinical deterioration event for clarity and conciseness. In other words, various embodiments described herein in context of methods (e.g., the method 100 of predicting a clinical deterioration event) are analogously valid for the corresponding systems or devices (e.g., the system 300 for predicting a clinical deterioration event), and vice versa.

[0053] FIG. 4 depicts a schematic block diagram of a system 400 for training a deterioration prediction machine learning model, according to various embodiments of the present invention. The system 400 comprises: at least one memory 402; and at least one processor 404 communicatively coupled to the at least one memory 402 and configured to perform the method 200 of training a deterioration prediction machine learning model for predicting a clinical deterioration event according to various embodiments of the present invention. Accordingly, the at least one processor 404 is configured to: obtain time-series data of vital signmeasurements of a subject with respect to a time interval defined by a start time point and an end time point; generate a training feature dataset comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject, wherein the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window to the end time point has a longer time period than the time period of the first time window and the training feature dataset comprises the first and second sets of vital sign statistical features; label the training feature dataset as a positive class or a negative class based on whether the time-series data of vital sign measurements is associated with a clinical deterioration event or not; and train the deterioration prediction machine learning model based on the labelled training feature dataset for predicting a clinical deterioration event for a subject.

[0054] It will be appreciated by a person skilled in the art that the at least one processor 404 may be configured to perform various functions or operations through set(s) of instructions (e.g., software modules) executable by the at least one processor 404 to perform various functions or operations. Accordingly, as shown in FIG. 4, the system 400 may comprise: a vital sign measurement obtaining module (or a vital sign measurement obtaining circuit) 406 configured to obtain time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point; a training feature dataset generating module (or a training feature dataset generating circuit) 408 configured to generate a training feature dataset comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject, wherein the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window to the end time point has a longer time period than the time period of the first time window; a labelling module (or a labelling circuit) 410 configured to label the training feature dataset as a positive class or a negative class based on whether the time-series data of vital sign measurements is associated with a clinical deterioration event or not; and a training module (or a training circuit) 412 configured to train the deterioration prediction machine learning model based on the labelled training feature dataset for predicting a clinical deterioration event for a subject (e.g., a patient).

[0055] Similarly, it will be appreciated by a person skilled in the art that the above-mentioned modules of the system 400 are not necessarily separate modules, and two or more modules may be realized by or implemented as one functional module (e.g., a circuit or a software program) as desired or as appropriate without deviating from the scope of the present invention. For example, two or more of the vital sign measurement obtaining module 406, the training feature dataset generating module 408, the labelling module 410 and the training module 412 may be realized (e.g., compiled together) as one executable software program (e.g., software application), which for example may be stored in the at least one memory 402 and executable by the at least one processor 404 to perform the corresponding functions or operations as described herein according to various embodiments of the present invention.

[0056] In various embodiments, the system 400 for training a deterioration prediction machine learning model corresponds to the method 200 of training a deterioration prediction machine learning model as described hereinbefore with reference to FIG. 2, therefore, various operations, functions or steps configured to be performed by the at least one processor 404 may correspond to various operations, functions or steps of the method 200 of training a deterioration prediction machine learning model described hereinbefore according to various embodiments, and thus need not be repeated with respect to the system 400 for training a deterioration prediction machine learning model for clarity and conciseness. In other words, various embodiments described herein in context of methods (e.g., the method 200 of training a deterioration prediction machine learning model) are analogously valid for the corresponding systems or devices (e.g., the system 400 for training a deterioration prediction machine learning model), and vice versa.

[0057] A computing system, a controller, a microcontroller or any other system providing a processing capability may be provided according to various embodiments in the present invention. Such a system may be taken to include one or more processors and one or more computer-readable storage mediums. For example, the system 300 for predicting a clinical deterioration event for a subject described hereinbefore includes at least one processor 304 and at least one computer-readable storage medium (or memory) 302 which are for example used in various processing carried out therein as described herein. A memory or computer-readable storage medium used in various embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory,an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory). Furthermore, it will be appreciated by a person skilled in the art that the system 400 for training a deterioration prediction machine learning model may be implemented by a high-performance computer system, or a network of high-performance computer systems, known in the art for performing training, especially when a large-scale training is performed. For example, the high-performance computer system may include multiple processors including GPUs (graphics processing units) and CPUs (central processing units) optimized for advanced computing tasks, such as execution of machine learning algorithms. For example, the high-performance computer system may comprise an array of GPUs (Graphics Processing Units) dedicated to handling parallel processing tasks to facilitate the rapid execution of machine learning algorithms. In the field of machine learning, it will be understood by a person skilled in the art that the system 400 for training a deterioration prediction machine learning model may be implemented as a server (e.g., centralized or distributed server (centralized or decentralized machine learning)), such as a cloud server, including specialized hardware (e.g., high performance graphics processing unit(s) (GPU(s))) designed for handling resourceintensive tasks associated with machine learning.

[0058] In various embodiments, a “circuit” may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, firmware, or any combination thereof. Thus, in an embodiment, a “circuit” may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g., a microprocessor (e.g., a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor). A “circuit” may also be a processor executing software, e.g., any kind of computer program, e.g., a computer program using a virtual machine code, e.g., Java. Any other kind of implementation of various functions or operations may also be understood as a “circuit” in accordance with various other embodiments. Similarly, a “module” may be a portion of a system according to various embodiments in the present invention and may encompass a “circuit” as above, or may be understood to be any kind of a logic-implementing entity therefrom.

[0059] Some portions of the present disclosure may be explicitly or implicitly presented in terms of algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means used by those skilled in the data processing arts to convey most effectively thesubstance of their work to others skilled in the art. An algorithm may be, and generally, conceived to be a self-consistent sequence of steps leading to a desired result.

[0060] The present specification also discloses systems (e.g., which may also be embodied as devices or apparatuses), such as the system 300 for predicting a clinical deterioration event for a subject and the system 400 for training a deterioration prediction machine learning model, for performing various operations, functions or steps of various methods described herein. Such systems may each be specially constructed for the required purposes or may comprise a general purpose computer system selectively activated or reconfigured by a computer program stored in the computer system. In general, various algorithms that may be presented herein are not limited to being implemented or executed by any particular computer system. Alternatively, the construction of more specialized computer system (e.g., a high-performance computer system as described hereinbefore) to perform various operations, functions or steps of various methods described herein may be provided as desired or as appropriate without going beyond the scope of the present invention.

[0061] In addition, the present specification also at least implicitly discloses computer program(s) or software / functional module(s), in that it would be apparent to a person skilled in the art that various operations, functions or steps of various methods described herein may be put into effect by computer code. The computer program(s) is not intended to be limited to any particular programming language and implementation thereof, and it will be appreciated by a person skilled in the art that a variety of programming languages and coding thereof may be used to implement the computer program(s). Moreover, the computer program(s) is not intended to be limited to any particular control flow as there are a variety of programming languages which can use different control flows. It will be appreciated by a person skilled in the art that a computer program may be stored on any computer-readable storage medium (non-transitory computer-readable storage medium), such as but not limited to, a magnetic disk, an optical disk or a memory chip. For example, a computer program stored on a computer-readable storage medium may be loaded and executed on a computer system to implement various operations, functions or steps of various methods described herein according to various embodiments of the present invention.

[0062] Accordingly, in various embodiments, there is provided a computer program product, embodied in one or more computer-readable storage mediums (non-transitory computer-readable storage medium), comprising instructions (e.g., the vital sign measurement obtaining module 306, the inference feature dataset generating module 308 and / or thedeterioration prediction module 310) executable by one or more computer processors to perform the method 100 of predicting a clinical deterioration event as described hereinbefore with reference to FIG. 1 according to various embodiments of the present invention. In various embodiments, there is provided a computer program product, embodied in one or more computer-readable storage mediums (non-transitory computer-readable storage medium), comprising instructions (e.g., vital sign measurement obtaining module 406, the training feature dataset generating module 408, the labelling module 410 and / or the training module 412) executable by one or more computer processors to perform the method 200 of training a deterioration prediction machine learning model as described hereinbefore with reference to FIG. 2 according to various embodiments of the present invention. Accordingly, various computer programs or software modules described herein may be stored in a computer program product receivable by a system therein for execution by at least one processor of the system to perform various operations, functions or steps of various methods described herein according to various embodiments of the present invention.

[0063] It will be appreciated by a person skilled in the art that various modules of systems described herein (e.g., the vital sign measurement obtaining module 306, the inference feature dataset generating module 308 and / or the deterioration prediction module 310 for the system 300 for predicting a clinical deterioration event and the vital sign measurement obtaining module 406, the training feature dataset generating module 408, the labelling module 410 and / or the training module 412 for the system 400 for training a deterioration prediction machine learning model) may be software module(s) realized by computer program(s) or set(s) of instructions executable by a computer processor to perform various functions or operations. Various modules described herein (e.g., the vital sign measurement obtaining module 306, the inference feature dataset generating module 308 and / or the deterioration prediction module 310 for the system 300 for predicting a clinical deterioration event and the vital sign measurement obtaining module 406, the training feature dataset generating module 408, the labelling module 410 and / or the training module 412 for the system 400 for training a deterioration prediction machine learning model) may also be implemented as hardware module(s) being functional hardware unit(s) designed to perform various functions or operations. More particularly, in the hardware sense, a module is a functional hardware unit designed for use with other components or modules. For example, a module may be implemented using discrete electronic components, or it can form a portion of an entire electronic circuit such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA). Numerous other possibilitiesexist. It will also be appreciated by a person skilled in the art that a combination of hardware and software modules may be implemented. Furthermore, various operations, functions or steps of various methods described herein may be performed in parallel rather than sequentially as desired or as appropriate (e.g., as long as it does not render the method(s) inoperable or unsatisfactory for its intended purpose).

[0064] It will be appreciated by a person skilled in the art that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0065] Any reference to an element or a feature herein using a designation such as “first”, “second” and so forth does not limit the quantity or order of such elements or features, unless stated or the context requires otherwise. For example, such designations may be used herein as a convenient way of distinguishing between two or more elements or instances of an element. Thus, a reference to first and second elements does not necessarily mean that only two elements can be employed, or that the first element must precede the second element, unless stated or the context requires otherwise. In addition, a phrase referring to “at least one of’ a list of items refers to any single item therein or any combination of two or more items therein.

[0066] In order that the present invention may be readily understood and put into practical effect, various example embodiments of the present invention will be described hereinafter by way of examples only and not limitations. It will be appreciated by a person skilled in the art that the present invention may, however, be embodied in various different forms or configurations and should not be construed as limited to the example embodiments set forth hereinafter. Rather, these example embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present invention to those skilled in the art.

[0067] Various example embodiments of the present invention provide a predictive model (a deterioration prediction machine learning model) to identify patients at risk of clinical deterioration using vital sign data (time-series data of vital sign measurements). In various example embodiments, the vital sign data comprises heart rate measurements, respiratory ratemeasurements, oxygen saturation measurements, and systolic blood pressure measurements. In various example embodiments, the predictive model is a significantly improved machine learning model or algorithm trained to predict a clinical deterioration event for a subject, which for example, may be applied for identifying intensive care unit (ICU) transfers and rapid response team (RRT) activations in hospitalized patients (e.g., adult patients).

[0068] As an illustrative example, an example predictive model is trained to predict a clinical deterioration event (e.g., resulting in a RRT call or an unplanned ICU transfer) using vital sign statistical features generated or extracted from time-series vital sign data, and more particularly, using four different types of vital sign measurements, namely, heart rate measurements, respiratory rate measurements, oxygen saturation measurements, and systolic blood pressure measurements. In the illustrative example, vital sign data from 227,858 inpatients admitted to general care or telemetry beds were obtained. After filtering for improving data quality to obtain high-quality data, data from 30,118 patients were used to train a Light Gradient Boosting Machine (LightGBM), and data from 30,095 patients were reserved for a blind validation. The trained predictive model was validated and validation results show that at a sensitivity of 73%, the trained predictive model achieved a positive predictive value (PPV) of 30%, with a C-statistic of 0.87, an alert rate of 0.17 per day and a normalized alert rate of 2.4. Stratified analysis by hospital revealed that PPV was highest at the Rochester site, reaching 55%. The example predictive model and the validation results will be described in more detail below according to various example embodiments of the present invention.

[0069] Achieving a high PPV is crucial, as it ensures a larger proportion of alerts are true positives, reducing the burden of false alarms. For example, in the above-mentioned illustrative example, the PPV of the primary model with four vital signs, as tested in Rochester site, outperforms the traditional deterioration index by 46% or a factor of 6.

[0070] Accordingly, as an illustrative example, a machine learning model is developed / trained according to various example embodiments utilizing four vital signs (i.e., four types of vital signs) to predict a clinical deterioration event (which may thus be referred to as a four-vital model), capable of providing early warnings of clinical deterioration with an enhanced PPV. It will be appreciated by a person skilled in the art that the present invention is not limited to a machine learning model utilizing four vital signs and that machine learning models may be provided utilizing more or fewer vital signs (i.e., more or fewer types of vital signs, such as temperature measurements) as desired or as appropriate (e.g., for better real-world applicability) without going beyond the scope of the present invention.

[0071] As discussed in the background, while early warning systems have been developed to predict clinical deterioration, they suffer from a low positive predictive value (PPV), leading to frequent false alarms. A low PPV will contribute to alarm fatigue among healthcare providers, with excessive alerts desensitizing staff and delaying responses to true emergencies. To address this technical problem, various example embodiments develop / train a machine learning model (a deterioration prediction machine learning model) with a focus on achieving an improved or high PPV for early prediction of clinical deterioration. By leveraging advanced feature extraction methods and time-series data from vital signs, the model aims to minimize false positives, ensuring that alerts are more actionable and reducing the burden on clinical staff while improving patient safety and outcomes. In this regard, various example embodiments seek to provide a predictive model to identify subjects / patients at risk of clinical deterioration, that is, predicting a clinical deterioration event (e.g., which may be defined as requiring a RRT intervention or an unplanned ICU transfer). In various example embodiments, the predictive model utilizes vital sign readings including heart rate, respiratory rate, oxygen saturation, and systolic blood pressure. In various example embodiments, the predictive model is advantageously applied to predict a clinical deterioration event on patients in general care settings, aiming to provide accurate, real-time predictions that assist healthcare providers in timely decision-making and early intervention to improve patient outcomes.

[0072] In the above-mentioned illustrative example, Mayo Clinic digital data from the Mayo Clinic Accelerate platform was used for training and validating the predictive model. The data sources were Mayo Clinic sites across Rochester, Florida, Arizona, and the Mayo Clinic Health System (MCHS). MCHS included 16 hospitals and 53 clinics in rural areas. The development focused on adult patients (aged 18 years or older) who were admitted to general care or telemetry beds. Vital sign data was extracted from inpatient admissions between January 1, 2019, and July 31, 2023. Hospitalizations were excluded if they consisted entirely of ICU stays or were primarily for research, rehabilitation, or psychiatric care. Longitudinal data from eligible patients were collected, including four types of vital signs, namely, respiration rate (RR), heart rate (HR), oxygen saturation (SpCh), and systolic blood pressure (SBP), as well as two types of demographic variables (demographic features): patient age and gender. These data served as the raw input for the predictive model to predict deterioration event(s).

[0073] Clinical deterioration events may occur multiple times within a single hospitalization. In various example embodiments, an episode is defined as a period (or time interval) during which a patient remained in a general care or telemetry bed withoutinterruptions caused by deterioration events or medical procedures. Furthermore, a deterioration event can only occur at the end of an episode (e.g., not in the middle of an episode). In other words, if / when a deterioration event occurs, the episode ends at the deterioration event. As a result, one hospitalization may give rise to multiple distinct episodes. In total, in the vital sign dataset obtained in the illustrative example, there were 227,858 unique patients with 444,582 unique episodes.

[0074] In various example embodiments, to enhance or ensure data quality, time points (measurement time points) where at least two of the four vital signs are missing are excluded. Additionally, episodes that have a time gap (time period) of six hours or more between consecutive vital sign measurements are removed. For example, this ensures that at least three vital signs are recorded within every six-hour period. Furthermore, for example, a patient who did not require vital sign monitoring for more than six hours can be assumed to be in a more stable condition. In the illustrative example, the demographics of patients with high-quality data used for training and validation are shown in Table 1 below. For model validation, approximately 50% of all episodes were set aside and kept entirely unseen during training. The remaining 50% was used for training the model and optimizing its hyperparameters. The data split was stratified to maintain a similar ratio of deterioration events between the training and validation sets. Importantly, the split was performed at the patient level to prevent any data leakage between the training and validation sets.Table 1 - Patient demographics" " ""<" <

[0075] FIG. 5 depicts a schematic drawing of an example method 500 of predicting a clinical deterioration event for a subject, according to various example embodiments of the present invention. According to the method 500, at 502, patient demographics (e.g., age and gender) and time-series data of vital sign measurements (e.g., a time-series episode (an episode of time-series data) of vital sign measurements) may be obtained as inputs. Before feature extraction, at 504, a data quality enhancing algorithm may be applied to enhance the reliability of the input data. Subsequently, at 506, vital sign statistical features may be extracted or generated and used as inputs for a machine learning model trained to predict (at 508) a clinical deterioration event for a subject. By way of an example only and without limitation, the selected model architecture may be Light Gradient Boosting Machine (LightGBM), which produces a final output indicating the probability of a clinical deterioration event for a patient.

[0076] In various example embodiments, at 504, the data quality enhancing algorithm may be configured to apply two filtering criteria to enhance data quality or integrity. According to a first filtering criterion, measurement time points where at least two of the four vital signs are missing are removed. According to a second (or resolution) filtering criterion, episodes that have a time gap of 6 hours or more between consecutive vital sign measurements (i.e., spaced 6 hours or more apart) are discarded. Furthermore, in various example embodiments, at 504, the data quality enhancing algorithm may be further configured to replace out-of-range (or unrealistic) vital sign values with missing values (e.g., NaN (Not a Number) values). For example, the LightGBM is able to handle NaN values. These out-of-range values are likely the result of instrument or human errors rather than accurate measurements. By replacing out-of-range vital sign values, the model does not receive these erroneous values, thereby enhancing the model’s predictions. Therefore, the data quality enhancing algorithm advantageously minimizes undesirable distortions and inconsistencies to model predictions that can negatively impact model predictions. By way of examples only and without limitations, example acceptable ranges for each of the four vital signs are shown in Table 2 below.Table 2 - Acceptable ranges for each vital sign"l ’ l

[0077] In various example embodiments, at 506, feature extraction may be performed using multiple time windows, such as the two example time windows 610, 620 as illustrated in FIG.6A. In particular, FIG. 6A shows a schematic drawing of time-series data of vital sign measurements of a subject obtained with respect to a time interval 604 defined by a start time point 606 and an end time point 608, whereby the end time point 608 is a prediction time point at which the clinical deterioration event for the subject is being predicted, with an example first time window 610 and an example second time window 620 illustrated. In this regard, an inference feature dataset comprising a first set and a second set of vital sign statistical features is generated (or extracted). The first set of vital sign statistical features is generated (or extracted) from the first time window 610 and the second set of vital sign statistical features is generated (or extracted) from the second time window 620 of the time-series data of vital sign measurements. In particular, the first time window 610 has a time period that ends at the end time point 608, the second time window 620 has a time period that starts at the start time point 606, and a first time period from an end 622 of the second time window 620 to the end time point 608 has a longer time period than the time period of the first time window 610. For example, the first time window 610 may be referred to as a current window (e.g., since more recent vital signal measurements are in the first time window 610, e.g., resulting in shorter-term vital signal trend features) and the second time window 620 may be referred to as a baseline window (e.g., since older vital sign measurements are in the second time window 620, e.g., resulting in longer-term vital sign trend features as vital sign statistical features over a longer duration or timeframe may be captured). Accordingly, the method 500 generates (or extracts) vital sign statistical features from at least two time windows of the time-series data of vital sign measurements covering different time segments or zones of the time interval 604. In particular, because the above-mentioned first time period is configured to have a longer time period (e.g., 4 to 48 times larger) than the time period of the first time window 610, the vital sign statistical features are generated from the first time window 610 (having a time period that ends at the end time point 608 of the time interval 604) based on more recent vital sign measurements (e.g., resulting in shorter-term vital sign trend features) compared to the vital sign statistical features generated from the second time window 620 (having a time period that starts at the start time point 606 of the time interval 604), which are based on older vital sign measurements (obtained longer ago) (e.g., resulting in longer-term vital sign trend features). This enables the method 500 to generate (or extract) vital sign statistical features over different time segments or zones (recent vs older), which has been surprisingly found to improve (e.g., significantly improve)accuracy in predicting a clinical deterioration event for the subject based on such vital sign statistical features generated.

[0078] Furthermore, since the above-mentioned first time period is configured to have a set (predefined) time period longer than the time period of the first time window 610, the time period or length of the second time window 620 grows over time as the time-series data of vital sign measurements (or the number of vital sign measurements) grows over time, thereby enabling a growing number of past vital sign measurements (starting from the start time point 606 of the time interval 604) to be captured or utilized to, for example, establish better baseline or longer-term trends (vital sign trend features). Accordingly, the method 500 may be performed iteratively over time at each new (or next) prediction time point 608 at which the clinical deterioration event for the subject is being predicted. In this regard, for each iteration for a new (or next) prediction time point 608, the time-series data of vital sign measurements of the subject is obtained with respect to a time interval defined (or bounded) by the start time point 606 and the end time point 608, the end time point 608 being the new prediction time point. In various example embodiments, the above-mentioned each new prediction time point corresponds to each new measurement time point. That is, the method 500 may be configured to predict a clinical deterioration event for the subject over time (e.g., until a clinical deterioration event is predicted or occurs) at each new measurement time point (i.e., each time vital sign measurements are obtained from the subject). Therefore, the method 500 may advantageously be employed to continuously or repeatedly monitor the subject for clinical deterioration event. Accordingly, both the first and second time windows 610, 620 may be referred to as running windows as the first (or current) time window 610 (having a predefined time period) moves / shifts over time with each new (or next) prediction time point 608 (or with each new measurement time point), and the second (or baseline) time window 620 grows in length since its end 622 has a predefined time period / gap (i.e., the above-mentioned first time period) to the end time point (the prediction time point) 608.

[0079] For generating (or extracting) vital sign statistical features at (or with respect to) a prediction time point (which may also be referred to as an inference time point) t (which corresponds to the end time point 608 shown in FIG. 6A)):• the current time window 610 extends from t - x (x hours (e.g., 1 hour illustrated in FIG.6A) prior to t) to t (the prediction time point 608); and• the baseline time window 620 extends from the start 606 of the time interval (e.g., the start / beginning of the episode, e.g., when a patient gets admitted at the 0thhour or timepoint (or returns from a deterioration event or medical procedure) or may correspond to a time point of a first vital sign measurements of the patient (a first measurement time point)) to (or t = 0) to t - y (ending y hours (corresponding to the above-mentioned first time period) prior t (the prediction time point 608)).

[0080] A combination or set of statistical features is extracted from each time window. As illustrated in FIG. 6A, the current time window 610 comprises a plurality of measurement time points (represented by filled circles / dots), and the baseline time window 620 comprises a plurality of measurement time points (represented by filled circles / dots). In this regard, the first set of vital sign statistical features is generated (or extracted) from the current time window 610 by determining the first set of vital sign statistical features based on vital sign measurements at the plurality of measurement time points of the current time window 610. Similarly, the second set of vital sign statistical features is generated (or extracted) from the baseline time window 620 by determining the second set of vital sign statistical features based on vital sign measurements at the plurality of measurement time points of the baseline time window 620.

[0081] In various example embodiments, the vital sign measurements comprise at least two different types of vital sign measurements of the subject. In this regard, for each type of vital sign measurements, each of the first and second sets of vital sign statistical features comprises a plurality of different types of vital sign statistical features derived from the type of vital sign measurements. In various example embodiments, the vital sign measurements comprise two or more of respiration rate measurements, heart rate measurements, oxygen saturation measurements, blood pressure measurements and temperature measurements of the subject. In the above-mentioned illustrative example, the vital sign measurements comprise four types of vital signs, namely, respiration rate (RR) measurements, heart rate (HR) measurements, oxygen saturation (SpCh) measurements, and systolic blood pressure (SBP) measurements. In various example embodiments, the first set of vital sign statistical features is a first set of vital sign trend features, and the second set of vital sign statistical features is a second set of vital sign trend features. In this regard, in various example embodiments, each of the plurality of different types of vital sign statistical features derived from the type of vital sign measurements is generated based on a mean, a standard deviation, a minimum, a maximum, a median, a quartile, an interquartile range, a mean absolute deviation, a variance or a mode of the type of vital sign measurements. For example, each of the plurality of different types of vital sign statistical features derived from the type of vital sign measurements is generated based on a mean, a standard deviation, a minimum, a maximum or a median of the type of vital sign measurements.

[0082] In various example embodiments, the above-mentioned first time period is about 4 to 48 times larger than the time period of the first / current time window 610. In this regard, the first time period may be set or optimized according to a desired or an optimized time separation or gap between the first / current and second / baseline time windows 610, 620, such as based on a desired or an optimized time difference between recent vital sign measurements (e.g., for generating shorter-term vital sign trend features) and older vital sign measurements (e.g., for generating longer-term vital sign trend features), for example, for optimizing trend differentiation. In various example embodiments, the first time period may be about 6 to 46 times larger, about 8 to 44 times larger, about 10 to 42 times larger, about 12 to 40 times larger, about 14 to 38 times larger, about 16 to 36 times larger, about 18 to 34 times larger, about 20 to 32 times larger, about 22 to 30 times larger, about 22 to 28 times larger, about 22 to 26 times larger or about 23 to 25 times larger than the time period of the first / current time window 610.

[0083] In various example embodiments, the time period of the current time window 610 is in a range of about 30 minutes to about 24 hours, and the above-mentioned first time period is in a range of about 4 hours to about 48 hours, whereby the above-mentioned first time period has a longer period than the time period of the current time window 610. In this regard, in various example embodiments, the time period of the first / current time window 610 may be in a range of about 30 minutes to about 20 hours, about 30 minutes to about 18 hours, about 35 minutes to about 16 hours, about 35 minutes to about 14 hours, about 40 minutes to about 12 hours, about 40 minutes to about 10 hours, about 40 minutes to about 8 hours, about 45 minutes to about 6 hours, about 45 minutes to about 4 hours, about 45 minutes to about 2 hours, about 45 minutes to about 90 minutes or about 50 minutes to about 70 minutes, and the above-mentioned first time period may be in a range of about 6 hours to about 46 hours, about 8 hours to about 44 hours, about 10 hours to about 42 hours, about 12 hours to about 40 hours, about 14 hours to about 38 hours, about 14 hours to about 38 hours, about 16 hours to about 36 hours, about 18 hours to about 34 hours, about 20 hours to about 32 hours, about 22 hours to about 30 hours, about 22 hours to about 28 hours, about 22 hours to about 26 hours or about 23 hours to about 25 hours.

[0084] Therefore, a first rolling time window 610 is utilized for each episode as a current time window for capturing statistical features with a time period in a range of about 30 minutes to about 24 hours prior to the prediction time point 608. To enrich the feature dataset, a second time window 620 is utilized as a baseline time window for capturing statistical features with an end 622 thereof to the prediction time point 608 having a time period or gap in a range of about4 hours to about 48 hours, while having a longer period than the time period of the current time window 610, which advantageously provides valuable baseline vital sign statistical features. For example, by choosing descriptive statistics for both the current time window 610 and the baseline time window 620, the model compares actual values and trends throughout an episode, helping the model better capture underlying patterns in the vital signs. According to various example embodiments, it is found that selecting a larger difference between the time period of the current time window 610 and the above-mentioned first time period enables the model to more effectively capture significant changes in trends.

[0085] In various example embodiments, parameter optimization is conducted (e.g., using a grid search) to determine an optimal configuration of the two time windows 610, 620. In a parameter optimization, a current time window 610 with x = 1 hour (i.e., having a time period of 1 hour) and a baseline window 620 withy = 24 hours (i.e., an end 622 thereof to the prediction time point 608 having a time period or gap of 24 hours) are determined to be optimal. In various example embodiments, the statistical features that are determined to be most effective are the median, minimum, and maximum for both time windows 610, 620.

[0086] Apart from vital signs, in various example embodiments, at 506, first measurement interval information and second measurement interval information are extracted from the current time window 610 and the baseline time window 620, respectively. The first and second measurement interval information are included in the inference feature dataset. The measurement interval information extracted from a time window comprises, for each pair of consecutive measurement time points in the time window, the time interval length between the pair of consecutive measurement time points. The measurement interval information may also be referred to as sampling time information. For example, the sampling time may correspond to or indicate a worry factor, whereby shorter intervals between measurements may indicate that doctors or nurses are more concerned about a particular patient, leading to more frequent vital sign monitoring / measurements. In this regard, to enhance the model’s predictive capabilities, sample time is introduced as a feature for facilitating the model to better account for temporal patterns and serve as a proxy or measure of the worry factor. For example, it is found that variations in sampling time can serve as a reliable indicator of patient deterioration for the model. Accordingly, various example embodiments advantageously provide a multiwindow feature extraction approach or process with a worry factor proxy for a machine learning model for identifying patient deterioration based on vital sign measurements. For example, four different types of vital signs may be provided (RR measurements, HR measurements, SpO?measurements, and SBP measurements) as primary features or predictors and the sampling time may be provided as a proxy for a worry factor. In particular, according to various example embodiments, the integration of a multi-window feature extraction process, combined with the use of sampling time as a proxy of a worry factor, provides a unique approach to address the technical problems discussed in the background. By analyzing multiple temporal windows of physiological data and incorporating the sampling time as an indicator of clinical concern, various example embodiments provide a more comprehensive and adaptive assessment of patient status, thereby facilitating earlier and more accurate identification of clinical deterioration of patients.

[0087] Accordingly, from the feature extraction at 506, an inference feature dataset (e.g., a feature vector) comprising the vital sign statistical features generated from the current and baseline time windows 610, 620 is obtained. In various example embodiments, the inference feature dataset may further comprise the first and second measurement interval information, as well as the vital sign measurements (i.e., raw vital sign measurements) at a measurement time point at or nearest to the prediction time point 608 and patient demographic data (e.g., age and gender). For example, in the case of the prediction time point 608 being a measurement time point as described hereinbefore according to various example embodiments, the vital sign measurements at the prediction time point 608 is included in the inference feature dataset. In the above-mentioned illustrative example, for the feature extraction at 506, the inference feature dataset has a total of 37 features (whereby some features were missing values) and is shown in FIG. 7. For example, missing values may be left unaltered, since, for example, tree-based models are naturally able to handle them during the training and prediction processes.

[0088] The inference feature dataset generated is then inputted to a trained machine learning model to predict (at 508) a clinical deterioration event for the subject. The trained machine learning model may then output (at 510) a prediction of the clinical deterioration event for the subject. In various example embodiments, predicting the clinical deterioration event comprises estimating a probability of the clinical deterioration event for the subject based on the inference feature dataset. For example, the trained machine learning model may be a light gradient boosting machine (LightGBM) model configured to provide (at 510) an output indicating the probability of the clinical deterioration event for the subject.

[0089] In various example embodiments, one or more additional sets of vital sign statistical features may be generated respectively from one or more additional time windows of the timeseries data of the vital sign measurements of the subject, and included in the inference featuredataset. In this regard, the multiple time windows have different time periods. Each additional set of vital sign statistical features may be generated in the same or similar manner as the first or second set of vital sign statistical features as described above according to various example embodiments of the present invention, and thus need not be repeated for clarity and conciseness. Therefore, according to various example embodiments of the present invention, additional time windows may be provided to further enrich the inference feature dataset and provide the model with a more comprehensive understanding or information of trend changes.

[0090] In various example embodiments, the one or more additional time windows comprise one or more pairs of additional time windows, each pair of additional time windows comprises a first additional time window and a second additional time window wherein the first additional time window has a time period that ends at the end time point, the second additional time window has a time period that starts at the start time point, and a second time period from an end of the second additional time window (i.e., the end opposite to a start of the second additional time window) to the end time point has a longer time period than the time period of the first additional time window. For illustrative purposes, FIG. 6B shows a schematic drawing of time-series data of vital sign measurements of a subject obtained with respect to a time interval 604 defined by a start time point 606 and an end time point 608, whereby the end time point 608 is a prediction time point at which the clinical deterioration event for the subject is being predicted, with an example first pair of time windows comprising the example first time window 610 and the example second time window 620 and an example additional pair of time windows comprising an example first additional time window 630 and an example second additional time window 640. It will be appreciated by a person skilled in the art that the present invention is not limited to one additional pair of time windows and any number of additional pairs of time windows may be implemented as desired or as appropriate. For the example additional pair of time windows, the first additional time window 630 has a time period that ends at the end time point 608, the second additional time window 640 has a time period that starts at the start time point 606, and a second time period from an end 642 of the second additional time window 640 to the end time point 608 has a longer time period than the time period of the first additional time window 630. In various example embodiments, similar to the first pair of time windows, the second time period may be about 4 to 48 times larger than the time period of the first additional time window 630. In this regard, similar to the above-mentioned first time period, the second time period may be set according to a desired or an optimized time separation or gap between the first and second additional time windows 630,640, such as based on a desired or an optimized time difference between recent vital sign measurements (e.g., for generating shorter-term vital sign trend features) and older vital sign measurements (e.g., for generating longer-term vital sign trend features), for example, for optimizing trend differentiation. In this regard, the first and second time periods may be optimized together. By way of an illustrative example only and without limitation, the first time window 610 may have a time period of 1 hour, the first additional time window 630 may have a time period of 2 hours, the second time window 620 may have an end 622 thereof to the prediction time point 608 set to have a time period or gap of 24 hours and the second additional time window 640 may have an end 642 thereof to the prediction time point 608 set to have a time period or gap of 48 hours. For example, multiple pairs of time windows may be implemented to provide multiple sets of shorter-term vital sign trend features and multiple sets of longer-term vital sign trend features. Parameter optimization may also be conducted to determine an optimal configuration of the additional time windows.

[0091] In various example embodiments, a different set of vital sign statistical features (i.e., based on a different combination of types (including a different number of types) of vital sign measurements) may be extracted from each time window. As an illustrative example, FIG. 6C illustrates a schematic drawing of the time-series data of vital sign measurements shown in FIG.6A but with a different set of vital sign statistical features extracted from each time window 610, 620, and more specifically, a first set of vital sign statistical features generated from the current time window 610 based on median and maximum and a second set of vital sign statistical features generated from the current time window 620 based on mean, standard deviation, minimum and maximum.

[0092] In addition, as described hereinbefore, the method 500 may be performed iteratively over time at each new (or next) prediction time point 608 at which the clinical deterioration event for the subject is being predicted. In this regard, new vital sign measurements at new measurement time points may be obtained over time. As described above, the above-mentioned each new prediction time point may correspond to each new measurement time point, whereby the method 500 may be configured to predict a clinical deterioration event for the subject over time (e.g., until a clinical deterioration event is predicted or occurs) at each new measurement time point (i.e., each time vital sign measurements are obtained from the subject). Therefore, the method 500 may advantageously be employed to continuously or repeatedly monitor the subject for clinical deterioration event. Accordingly, both the first and second time windows may be referred to as running windows as the first (or current) time window 610 (having apredefined time period) moves / shifts over time with each new (or next) prediction time point 608, and the second (or baseline) time window 620 grows in length (thus may include more vital sign measurements at measurement time points) since its end 622 has a predefined time period / gap (i.e., the above-mentioned first time period) to the end time point (the prediction time point) 608, as illustrated in FIG. 6D. Accordingly, the time period of the first / current time window 610 and the first time period are fixed or constant.

[0093] There is also provided a corresponding method of training a deterioration prediction machine learning model for predicting a clinical deterioration event, according to various example embodiments of the present invention. The method comprising: obtaining time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point; generating a training feature dataset (or feature vector) comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject, wherein the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window to the end time point has a longer time period than the time period of the first time window, and the training feature dataset comprises the first and second sets of vital sign statistical features; labelling the training feature dataset as a positive class or a negative class based on whether the time-series data of vital sign measurements is associated with a clinical deterioration event or not; and training the deterioration prediction machine learning model based on the labelled training feature dataset for predicting a clinical deterioration event for a subject. In various example embodiments, the training feature dataset may be labelled as a positive class or a negative class based on whether the time-series data of vital sign measurements is associated with a clinical deterioration event as defined or not. It will be appreciated by a person skilled in the art that the method of training the deterioration prediction machine learning model may be performed for each time-series data of vital sign measurements obtained.

[0094] Since the method of training a deterioration prediction machine learning model corresponds to the method of predicting a clinical deterioration event using a deterioration prediction machine learning model as described hereinbefore according to various example embodiments of the present invention, it will be appreciated by a person skilled in the art that the first and second time windows (as well as any additional time window(s)) in the method of training a deterioration prediction machine learning model may be configured in the same orsimilar manner as the first and second time windows 610, 620 (as well as any additional time window(s)) as described hereinbefore for the method 500 of predicting a clinical deterioration event according to various example embodiments of the present invention, and thus need to be repeated for clarity and conciseness. That is, the training feature dataset may be generated or obtained in the same or similar manner as the inference feature dataset. Similarly, it will be appreciated by a person skilled in the art that the training feature dataset (e.g., the first and second sets of vital sign statistical features (as well as any additional set(s) of vital sign statistical features, any other features of the training feature dataset)) may be generated in the method of training a deterioration prediction machine learning model in the same or similar manner as the inference feature dataset (e.g., the first and second sets of vital sign statistical features (as well as any additional set(s) of vital sign statistical features, any other features of the feature dataset)) generated in the method 500 of predicting a clinical deterioration event as described hereinbefore according to various example embodiments of the present invention, and thus need to be repeated for clarity and conciseness.

[0095] In various example embodiments, the time-series data of vital sign measurements is associated with a clinical deterioration event and the end time point corresponds to a start (or an onset) of the clinical deterioration event. In various example embodiments, the time-series data of vital sign measurements is not associated with a clinical deterioration event and the end time point is a time point randomly selected in the time interval.

[0096] In the above-mentioned illustrative example, the deterioration prediction machine learning model used is a gradient boosting tree-based algorithm referred to as LightGBM. For episodes with a deterioration event at an end thereof, a training feature dataset (feature vector) generated with respect to an end time point of a time interval (or an episode) (the end time point being at the start of the deterioration event) is selected for training as a positive class. For episodes without a deterioration event, a training feature dataset (feature vector) is generated with respect to a time point of a time interval (or an episode) which may be randomly selected (e.g., any measurement time point) within the time interval for representing a negative class (for training as a negative class). In various example embodiments, hyperparameter optimization is performed for the model using a grid search with 10-fold cross-validation, aiming to maximize the average precision score (the area under the precision-recall curve). In the above-mentioned illustrative example, since the training data is imbalanced, with approximately a 1 : 10 ratio of positive to negative classes, a higher penalty is applied to incorrectpredictions of the positive class. Specifically, the penalty for misclassifying a positive instance was scaled to be 10 times greater than that for a negative instance.

[0097] An illustrative validation technique or protocol will now be described according to various example embodiments of the present invention. In the above-mentioned illustrative example, the deterioration prediction machine learning model serves as a predictive tool for estimating the probability of a clinical deterioration event (which may be predefined as desired or as appropriate) occurring within the next 24 hours. To ensure robust evaluation, the validation set was kept separate and used exclusively for validation, with no overlap of patients between the training and validation sets. Descriptive statistics for both the training and validation sets are presented in Table 3 below.Table 3 - Descriptive statistics of the validation set and the train set"

[0098] To validate the deterioration prediction machine learning model in practical use, an alert generation technique or algorithm is designed to notify caregivers of potential deterioration events based on the model’s predictions. An alert is triggered when the predicted probability exceeds a predefined threshold. Once triggered, no further alerts are allowed within the prediction period, which is set to 24 hours as an illustrative example. Using the alert generation algorithm, the model’s performance can be evaluated on the validation set as follows:• a true positive (or true alert, TP) is an alert that is followed by a deterioration event within 24 hours;• a false positive (or false alert, FP) is an alert that is not followed by a deterioration event within 24 hours;• a false negative (FN) is a deterioration event that is not preceded by an alert within the prior 24 hours; and• a true negative (TN) is a number of days (multiple of 24 hours) with no deteriorationevents and no alerts triggered in the preceding 24 hours.

[0099] Every measurement time point within the episodes was processed by the model, and the predictive probability for each measurement time point was calculated. Based on the alert generation algorithm, an alert is either triggered or not at each measurement time point, depending on the model’s predicted probability of the deterioration event occurring within the next 24 hours.

[0100] With the four types of validation results explained above (TP, FP, FN, TN), the metrics computed include sensitivity (recall), specificity, positive predictive value (PPV or precision), and alert rate. While the ideal values for sensitivity, specificity, and PPV are 1.0 and are independent of the dataset, the ideal value of alert rate depends on the dataset. Specifically, the ideal alert rate occurs when all alerts are true positives, calculated as the total number of deterioration events in the dataset divided by the total length of all episodes (in days).

[0101] To better assess the model’s impact on alert rate, it may be insufficient to rely solely on the absolute value. Instead, a normalized value is considered, namely, the ratio of the model’s alert rate to the ideal alert rate, which reflects how many times more alerts the model generates compared to the ideal scenario. The derivation of this normalized alert rate shows that it is equivalent to the ratio of sensitivity to PPV. The normalized alert rate is derived as follows. The ideal alert rate is the ratio of number of positive events to the total length of all episodes in days: PE (number of positive events) / T (total length of all episodes in days). The model’s alert rate is the ratio of total number of alerts to the total length of all episodes in days: (TP (true positive) + FP (false positive)) / T. By definition, PPV (positive predictive value) = TP / (TP + FP), or TP + FP = TP / PPV. SE (sensitivity) = TP / PE, or TP = SE x PE, therefore TP + FP = SE x PE / PPV. The model’s alert rate is (SE x PE / PPV) / T = (sensitivity / PPV) x (PE / T). Finally, the normalized alert rate (the ratio of model’s alert rate to the ideal alert rate) is SE / PPV.

[0102] It is important to note that the normalized alert rate close to 1.0 does not always indicate good model performance, as it only implies that sensitivity and PPV are similar. This could mean the number of alerts per day is close to the ideal value, but the model may still generate many false alerts while missing true ones. Therefore, according to various example embodiments, the normalized alert rate is evaluated alongside other metrics, such as sensitivity and PPV at fixed sensitivity levels, to better assess the model’s performance.

[0103] In practice, there may be circumstances where not all vital signs are available for use as input data. To address this, the impact of reduced input data is evaluated by developingthree example comparative (or alternative) models, each utilizing a different subset of vital signs. As illustrative examples, a three-vital model is provided using RR, HR, and SpO2 as inputs and a two-vital model is provided using RR and HR, while a single-vital model is provided using only RR. Each of these models was trained and validated following the same or similar methodology as described hereinbefore for the example four-vital model.

[0104] Using the validation protocol described hereinbefore, the example four-vital model demonstrated strong performance. At a sensitivity of 73%, it achieved a specificity of 88%, a positive predictive value (PPV) of 30%, an area under the receiver operating characteristic curve (AUC-ROC, or C-statistic) of 0.87 and an average precision (AP or AUC under PR curve) of 0.48. The four-vital model also produced an alert rate of 0.17 alerts per day and a normalized alert rate of 2.4. For benchmarking purposes, the Deterioration Index (DI) was extracted from Mayo Clinic digital data within the Mayo Clinic Accelerate Platform and compared the model’s predictions to DI. The receiver operating characteristic (ROC) curve, the precision-recall curve (positive predictive value vs sensitivity) and the normalized alert rate of the four-vital model and the DI are shown in FIG. 8.

[0105] The four-vital model is also compared to an existing model designed for the same purpose, the Mayo Clinic Early Warning Score (MC-EWS) (Romero-Brufau S, et al., Using machine learning to improve the accuracy of patient deterioration predictions: Mayo Clinic Early Warning Score (MC-EWS). J Am Med Inform Assoc. 2021;28(6): 1207-1215). At a sensitivity of 73%, the results, summarized in Table 4 below, show that the four-vital model achieved a higher PPV with an 18% improvement. However, it had a 6% lower specificity and the alert rate that was higher by 0.1 alerts per day. The best performance was achieved when validated on Rochester data, with PPV of 55%. It is important to note that the four-vital model and MC-EWS were validated on different datasets and two metrics - alert rate and specificity are strongly influenced by the length of the dataset. Alert rate is inversely proportional to the total duration of the time series in the dataset, while specificity depends on true negative counts, which are also influenced by data length under the validation protocol.

[0106] In contrast, PPV and sensitivity are more comparable metrics because they rely on true positive, false positive, and false negative counts, which are unaffected by dataset length. A normalized alert rate is also more comparable, as it is adjusted by the ideal alert rate, making it independent of dataset length. Additionally, the deterioration event ratio in the dataset was 10.9%, compared to 12.1% in the MC-EWS dataset, placing both within a similar range and supporting the validity of PPV comparisons.Table 4 - Performance of the four-vital sign model, deterioration index and MC-EWSpy . . . .

[0107] The four-vital sign model’s performance was stratified by hospital site and patient race to assess potential effect measure modification. The ROC curves, PR curves and normalized alert rate for these stratifications are presented in FIGs. 9 A and 9B. In particular, FIGs. 9A and 9B show the ROC curves, PR curves and normalized alert rate of the four-vital sign model stratified by hospital site (FIG. 9A) and race (FIG. 9B). Among the hospital sites, the Rochester site demonstrated the highest positive predictive value (PPV) at a sensitivity of 0.73, achieving a PPV of 0.55, while the Florida site recorded the lowest PPV at 0.09. When stratified by race, the “Other” category exhibited the highest PPV at a sensitivity of 0.73, with a PPV of 0.45, whereas the lowest PPV was observed in the Black patient group, at 0.19.

[0108] The performance of the above-mentioned comparative models using subsets of vital signs, namely, three vitals (RR, HR, SpO2) (the three-vitals model), two vitals (RR, HR) (the two-vitals model), and a single vital (RR) (the one-vital model), is presented in FIG. 10, alongside the four-vital model, which utilized all four-vitals (RR, HR, SpO2, and SBP). In particular, FIG. 10 shows the ROC curve, PR curve and normalized alert rate of the one-vital, two-vitals, three-vitals and four-vitals models.

[0109] At a sensitivity of 0.73, the positive predictive value (PPV) decreased as the number of input vitals was reduced. The four-vitals model achieved a PPV of 0.30. In comparison, the comparative models achieved PPVs of 0.22, 0.20, and 0.18 for the three-vitals, two-vitals, and one-vital model, respectively.

[0110] In systems design to provide early warnings for patient deterioration, achieving a high positive predictive value (PPV) is both challenging and critically important. A higher PPV indicates that a larger proportion of alerts generated by the model are true positives, which is important for reducing the burden of false alarms. In real -world clinical settings, excessive false alarms can lead to alarm fatigue among healthcare providers, diminishing their ability to respond effectively to critical events. Therefore, optimizing PPV is not only a measure of themodel’s predictive accuracy but also a key factor in ensuring its practicality and adoption in clinical workflows.

[0111] For example, the example four-vitals model according to various example embodiments of the present invention achieved a PPV of 0.30, which is 3.75 times higher than the DI (PPV = 0.08) at the same level of sensitivity. Additionally, the alert rate generated by the four-vitals model are reduced fourfold, from 0.68 alerts per day to 0.17 alerts per day with the normalized alert rate reduced from 9.1 to 2.4 Although the DI was calculated using the same set of vital signs, the example four-vitals model leverages time-series statistical features extracted through running windows (current window and baseline window, as described hereinbefore). This approach allows the four-vitals model to incorporate more contextual information beyond the immediate vital signs, enabling it to detect patterns, learn from them, and ultimately deliver more precise predictions.

[0112] Several related studies have explored a variety of features and machine learning architectures to predict patient deterioration. For example, the four-vitals model is compared to MC-EWS and found that it achieved 2.5 times higher PPV but at the cost of lower specificity, although similar in range. When normalized by the ideal alert rate, the four-vitals model generated 40% fewer false alerts than MC-EWS at the same sensitivity level.

[0113] A key aspect in the method of predicting a clinical deterioration event for a subject according to various example embodiments of the present invention having better performance lies in the feature extraction approach wherein multiple aggregation time windows are used, such as the first (or current) time window 610 and the second (or baseline) time window 620. In various example embodiments, the current time window 610 represents a time period immediately preceding the prediction, while the baseline time window 620 covers a time period from admission up to a specified period prior to the prediction time.

[0114] As described hereinbefore according to various example embodiments of the present invention, the durations of these time windows may be fine-tuned during model optimization, with example optimal values being determined to be 1 hour for the first / current time window 610 and 24 hours for the above-mentioned first time period from and end 622 of the baseline time window to the prediction time point 608, as illustrated in FIG. 6A. This approach suggests that the most informative features are derived from recent data within the past hour, compared against baseline trends over the preceding 24 hours. Furthermore, according to various example embodiments, it is found that including raw vital signs at the current time significantly improved the model’s performance. Excluding these raw values during fine-tuning may resultin a significant drop in accuracy, emphasizing their complementary role alongside the statistical features.

[0115] An additional feature, apart from vital signs, according to various example embodiments of the present invention is measurement interval information associated with the time windows, which may also be referred to as sampling time. The measurement interval information is found to significantly improve model performance during development. For example, the sampling time can indirectly reflect a “worry factor” whereby shorter intervals between measurements may indicate that doctors or nurses are more concerned about a particular patient, leading to more frequent vital sign monitoring.

[0116] Another key feature according to various example embodiments of the present invention is leveraging a machine learning model’s (e.g., LightGBM’s) capability to handle missing values. Instead of imputing missing data with commonly used techniques, such as forward filling or running mean, missing data are left as they were, allowing the model to process the missingness directly. This approach not only simplified the preprocessing pipeline but also contributed to improved model performance.

[0117] The hospital-stratified results revealed significant performance differences across sites. The Rochester site outperformed the others, followed by Arizona, MCHS, and Florida. Notably, the PPV at the Rochester site is 0.46 higher than that at the Florida site, which recorded the lowest performance. Rochester accounted for approximately 40% of the data (see Table 1 above), making it possible for the model to overfit to this site. However, the other three hospitals each contributes roughly 20% of the data, yet their performances vary widely. Arizona achieved a PPV of 0.32, whereas Florida and MCHS had PPVs of 0.09 and 0.13, respectively. This discrepancy suggests that the observed performance differences are not purely due to overfitting; otherwise, the results across Arizona, MCHS, and Florida would be more consistent.

[0118] To better understand these disparities, a statistical analysis of the data was performed from each hospital, as shown in FIG. 11. In particular, FIG. 11 shows descriptive statistics of the validation set and the train set stratified by hospital. Several key differences between hospitals are identified. First, the ratio of deterioration events varies substantially. Florida and MCHS have deterioration ratios of approximately 6% to 7%, while Rochester’s ratio is nearly twice as high at 12%, and Arizona has the highest ratio at 17%, nearly three times higher than Florida. These variations could stem from differences in hospital practices, such ascriteria for calling a Rapid Response Team (RRT) or transferring patients to the ICU, or from variations in patient demographics and characteristics across hospitals.

[0119] Age distributions also differ across sites, particularly in specific age ranges. While the proportion of patients aged 60-80 years is similar across hospitals (48%- 51%), the proportion of patients under 60 years old ranges from 29% at MCHS to 40% at Rochester. Additionally, the proportion of patients over 80 years old is highest at MCHS (21%) and lowest at Rochester (11%). These demographic differences could influence model performance since the characteristics of the patient populations were not uniform across sites.

[0120] Another notable factor is the difference in sampling time distributions, which reflect variations in standard operating procedures. At Rochester, 75% of vital signs were sampled at intervals of 1 hour or less, while at Arizona, the 75th percentile of sampling intervals was approximately 3 hours, meaning vital signs were recorded three times less frequently compared to Rochester. Intuitively, more frequent vital sign sampling might improve model performance by providing more data for feature aggregation. While Rochester, with its frequent sampling, showed the best performance, Arizona achieved the second-best performance despite having the least frequent sampling intervals. This discrepancy may be attributed to Arizona’s higher proportion of deterioration events, although this hypothesis is not conclusive.

[0121] These findings underscore the complexity of hospital-level differences and their impact on model performance. Factors such as event ratios, patient demographics, and operational procedures all likely contribute to the observed variability, highlighting the need for further investigation and potentially site-specific optimization strategies.

[0122] The results of race-stratified analyses indicate that the majority of patients in the dataset are White, comprising approximately 90% of the population. At a sensitivity of 0.73, the PPV for White patients was 0.31, which closely aligned with the overall model performance of PPV 0.30. This suggests that the model’s (the four-vitals model’s) overall performance is heavily influenced by the White patient population due to its large proportion in the dataset. Notably, the “Other” and Asian racial categories outperformed the overall PPV, achieving 0.45 and 0.36, respectively. There was a significant performance gap between the best-performing category, “Other” (PPV = 0.45), and the worst-performing category, Black (PPV = 0.19). These discrepancies may stem from meaningful differences in vital sign patterns across races or variations in data collection practices and operational procedures. However, it is important to note that all non- White racial categories account for approximately 2% of the dataset each. Suchsmall sample sizes may lead to fluctuations in performance metrics, as the model predominantly learned from the characteristics of the White population.

[0123] Regarding the comparative models (the three-vitals, two-vitals, and one-vital models), their performances are predictably lower than the primary model since they utilize fewer vital signs, thereby reducing the amount of available information. Interestingly, the PPVs of the three-vital, two-vital, and one-vital models differed by approximately 0.02 from each other, whereas the primary model with four vital signs outperformed the three-vital model by a more substantial margin of 0.08 when SBP (systolic blood pressure) was included. This highlights the importance of SBP as a key vital sign for predicting deterioration.

[0124] Accordingly, the four-vitals model, developed using four vital signs (RR, HR, SpO2, and SBP) according to various example embodiments of the present invention demonstrated a PPV of 55% that was 6 times higher than the DI with the normalized alert rate of 1.3 on Rochester data. This improvement is attributed to an advanced feature extraction approach that incorporates multiple time windows (e.g., both a baseline window and a current window), which may be further improved by utilizing sampling time, which reflects the worry factor. For example, the baseline window may capture data trends over the previous 24 hours, while the current window may focus on the most recent hour, allowing the model to incorporate both historical and recent patient information effectively.

[0125] While embodiments of the invention have been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

1. CLAIMS1. A method of predicting a clinical deterioration event for a subject using at least one processor, the method comprising:obtaining time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point, wherein the end time point is a prediction time point at which the clinical deterioration event for the subject is being predicted;generating an inference feature dataset comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject, wherein the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window to the end time point has a longer time period than the time period of the first time window, and the inference feature dataset comprises the first and second sets of vital sign statistical features; andpredicting, using a deterioration prediction machine learning model, a clinical deterioration event for the subject based on the inference feature dataset.

2. The method according to claim 1, wherein the first time period is about 4 to 48 times larger than the time period of the first time window.

3. The method according to claim 1 or 2, wherein the time period of the first time window is in a range of about 30 minutes to about 24 hours, and the first time period is in a range of about 4 hours to about 48 hours.

4. The method according to claim 3, wherein the time period of the first time window is about 1 hour, and the first time period is about 24 hours.

5. The method according to any one of claims 1 to 4, wherein the first time window comprises one or more measurement time points, and the second time window comprises a plurality of measurement time points.

6. The method according to claim 5, whereinsaid generating the first set of vital sign statistical features from the first time window comprises determining the first set of vital sign statistical features based on vital sign measurements at the one or more measurement time points of the first time window, and said generating the second set of vital sign statistical features from the second time window comprises determining the second set of vital sign statistical features based on vital sign measurements at the plurality of measurement time points of the second time window.

7. The method according to claim 6, wherein the first set of vital sign statistical features is a first set of vital sign trend features, and the second set of vital sign statistical features is a second set of vital sign trend features.

8. The method according to any one of claims 1 to 7, whereinthe vital sign measurements comprise at least two different types of vital sign measurements of the subject, andfor each type of vital sign measurements, each of the first and second sets of vital sign statistical features comprises a plurality of different types of vital sign statistical features derived from the type of vital sign measurements.

9. The method according to claim 8, wherein the vital sign measurements comprise at least four different types of vital sign measurements of the subject.

10. The method according to any one of claims 1 to 9, wherein the vital sign measurements comprise two or more of respiration rate measurements, heart rate measurements, oxygen saturation measurements, blood pressure measurements and temperature measurements of the subject.

11. The method according to any one of claims 8 to 10, wherein each of the plurality of different types of vital sign statistical features derived from the type of vital sign measurements is generated based on a mean, a standard deviation, a minimum, a maximum, a median, a quartile, an interquartile range, a mean absolute deviation, a variance or a mode of the type of vital sign measurements.

12. The method according to any one of claims 1 to 11, wherein said generating the inference feature dataset further comprises extracting first measurement interval information from the first time window and second measurement interval information from the second time window, and the inference feature dataset further comprises the first and second measurement interval information.

13. The method according to any one of claims 1 to 12, wherein the inference feature dataset further comprises the vital sign measurements at a measurement time point at or nearest to the prediction time point.

14. The method according to any one of claims 1 to 13, wherein said predicting the clinical deterioration event comprises estimating, using the deterioration prediction machine learning model, a probability of the clinical deterioration event for the subject based on the inference feature dataset.

15. The method according to any one of claims 1 to 14, whereinsaid generating the inference feature dataset further comprises generating one or more additional sets of vital sign statistical features respectively from one or more additional time windows of the time-series data of the vital sign measurements of the subject,the first time window, the second time window, and the one or more additional time windows have different time periods, andthe inference feature dataset further comprises the one or more additional sets of vital sign statistical features.

16. The method according to any one of claims 1 to 15, whereinthe method is performed iteratively over time at each new prediction time point at which the clinical deterioration event for the subject is being predicted, andfor each iteration for a new prediction time point, the time-series data of vital sign measurements of the subject is obtained with respect to a time interval defined by the start time point and the end time point, the end time point being the new prediction time point.

17. A method of training a deterioration prediction machine learning model using at least one processor, the method comprising:obtaining time-series data of vital sign measurements of a subject with respect to a time interval defined by a start time point and an end time point;generating a training feature dataset comprising generating a first set of vital sign statistical features from a first time window and a second set of vital sign statistical features from a second time window of the time-series data of vital sign measurements of the subject, wherein the first time window has a time period that ends at the end time point, the second time window has a time period that starts at the start time point, and a first time period from an end of the second time window to the end time point has a longer time period than the time period of the first time window, and the training feature dataset comprises the first and second sets of vital sign statistical features;labelling the training feature dataset as a positive class or a negative class based on whether the time-series data of vital sign measurements is associated with a clinical deterioration event or not; andtraining the deterioration prediction machine learning model based on the labelled training feature dataset for predicting a clinical deterioration event for a subject.

18. The method according to claim 17, wherein the first time period is about 4 to 48 times larger than the time period of the first time window.

19. The method according to claim 17 or 18, wherein the time period of the first time window is in a range of about 30 minutes to about 24 hours, and the first time period is in a range of about 4 hours to about 48 hours.

20. The method according to claim 19, wherein the time period of the first time window is about 1 hour, and the first time period is about 24 hours.

21. The method according to any one of claims 17 to 20, wherein the first time window comprises one or more measurement time points, and the second time window comprises a plurality of measurement time points.

22. The method according to claim 21, whereinsaid generating the first set of vital sign statistical features from the first time window comprises determining the first set of vital sign statistical features based on vital sign measurements at the one or more measurement time points of the first time window, and said generating the second set of vital sign statistical features from the second time window comprises determining the second set of vital sign statistical features based on vital sign measurements at the plurality of measurement time points of the second time window.

23. The method according to claim 22, wherein the first set of vital sign statistical features is a first set of vital sign trend features, and the second set of vital sign statistical features is a second set of vital sign trend features.

24. The method according to any one of claims 17 to 23, whereinthe vital sign measurements comprise at least two different types of vital sign measurements of the subject, andfor each type of vital sign measurements, each of the first and second sets of vital sign statistical features comprises a plurality of different types of vital sign statistical features derived from the type of vital sign measurements.

25. The method according to claim 24, wherein the vital sign measurements comprise at least four different types of vital sign measurements of the subject.

26. The method according to any one of claims 17 to 25, wherein the vital sign measurements comprise two or more of respiration rate measurements, heart rate measurements, oxygen saturation measurements, blood pressure measurements and temperature measurements of the subject.

27. The method according to any one of claims 24 to 26, wherein each of the plurality of different types of vital sign statistical features derived from the type of vital sign measurements is generated based on a mean, a standard deviation, a minimum, a maximum, a median, a quartile, an interquartile range, a mean absolute deviation, a variance or a mode of the type of vital sign measurements.

28. The method according to any one of claims 17 to 27, wherein said generating the training feature dataset further comprises extracting first measurement interval information from the first time window and second measurement interval information from the second time window, and the training feature dataset further comprises the first and second measurement interval information.

29. The method according to any one of claims 17 to 28, wherein the training feature dataset further comprises the vital sign measurements at a measurement time point at or nearest to the end time point.

30. The method according to any one of claims 17 to 29, wherein said predicting the clinical deterioration event for the subject comprises estimating a probability of the clinical deterioration event for the subject.

31. The method according to any one of claims 17 to 30, whereinsaid generating the training feature dataset further comprises generating one or more additional sets of vital sign statistical features respectively from one or more additional time windows of the time-series data of the vital sign measurements of the subject,the first time window, the second time window, and the one or more additional time windows have different time periods, andthe training feature dataset further comprises the one or more additional sets of vital sign statistical features.

32. The method according to any one of claims 17 to 31, whereinthe time-series data of vital sign measurements is associated with a clinical deterioration event and the end time point corresponds to a start of the clinical deterioration event; or the time-series data of vital sign measurements is not associated with a clinical deterioration event and the end time point is a time point randomly selected in the time interval.

33. The method according to any one of claims 1 to 16, wherein the deterioration prediction machine learning model is trained according to any one of claims 17 to 32.

34. A system for predicting a clinical deterioration event for a subject, the system comprising:at least one memory; andat least one processor communicatively coupled to the at least one memory and configured to perform the method of predicting a clinical deterioration event for a subject according to any one of claims 1 to 16 and 33.

35. A system for training a deterioration prediction machine learning model, the system comprising:at least one memory; andat least one processor communicatively coupled to the at least one memory and configured to perform the method of training a deterioration prediction machine learning model according to any one of claims 17 to 32.

36. A computer program product, embodied in one or more non- transitory computer-readable storage mediums, comprising instructions executable by at least one processor to perform the method of predicting a clinical deterioration event for a subject according to any one of claims 1 to 16 and 33.

37. A computer program product, embodied in one or more non- transitory computer-readable storage mediums, comprising instructions executable by at least one processor to perform the method of training a deterioration prediction machine learning model according to any one of claims 17 to 32.

Images