Systems and methods for automation for machine-learning
The automation of machine-learning model retraining addresses data drift issues by automatically selecting optimal training data and models, ensuring continuous and efficient asset performance monitoring.
Patent Information
- Application Number
- PCT/US2026/027392
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-05-08
- Publication Date
- 2026-11-12
AI Technical Summary
Existing asset performance monitoring methods using machine-learning models are hindered by data drift, requiring manual intervention and expert judgment for retraining, which is inefficient and not suitable for real-time adjustments.
A computer-implemented method and system for automating the retraining of machine-learning models by detecting data drift, selecting optimal training data, and choosing the best model and hyperparameters, eliminating the need for manual intervention.
Enables continuous, automated retraining of machine-learning models to adapt to changing operational conditions, ensuring accurate and efficient performance monitoring without relying on human expertise.
Smart Images

Figure US2026027392_12112026_PF_FP_ABST
Abstract
Description
Docket No. 1086.2107001Systems and Methods for Automation for Machine-LearningRELATED APPLICATION
[0001] This application claims the benefit of U.S. Provisional Application No.63 / 802,892, filed on May 9, 2025. The entire teachings of the above application are incorporated herein by reference.BACKGROUND
[0002] Machine-learning (ML) is a subset of artificial intelligence (Al) directed to machines learning from data without explicit programming. A ML model is a method(s) that can find patterns or make decisions or predictions from previously unseen data. ML models learn from data through a process called training, in which such models are fed examples of data and iteratively adjust to minimize errors in their predictions.SUMMARY
[0003] Industrial asset performance monitoring may include deployment of methods, / .<?., machine-learning (ML) model(s), that process data from equipment sensors and allow a constant (continuous) assessment of health of such equipment. Asset performance monitoring methods can include anomaly detection, fault detection, regime discovery, and other types of assessments of a state of a system for non-limiting examples. In the state-of-the-art, such asset performance monitoring methods are trained from historical sensor data, manually selected by an expert, and then deployed to make live (real-time) predictions of potential asset performance for incoming data.
[0004] Real-time process data may, however, drift (deviate) away from process data that the asset performance monitoring methods, / .<?., ML model(s), have been trained on due to a variety of reasons, in which case, retraining the asset performance monitoring methods is useful. The present disclosure introduces a novel workflow to automate such retraining. Example embodiments disclosed herein may automatically select optimal sections of training data, automatically select an optimal ML model, and / or automatically detect a need to retrain the ML model, in which case, an example embodiment of a workflow disclosed herein may start over. An example embodiment disclosed herein enables constant (continuous)- 1 - 5083032. vlDocketNo. 1086.2107001deployment of agents (ML models) based on the best models for an actual mode of operation and avoids manual intervention used in the state-of-the-art performance monitoring methods.
[0005] According to an example embodiment, a computer-implemented method for automating ML model retraining detection may comprise receiving sensor data from at least one sensor coupled to a real-world system. The sensor data may be employed by a ML model trained to monitor performance of the real-world system. A current output of the ML model, determined by the ML model based on processing the sensor data received, may indicate current performance of the real-world system. The computer-implemented method may further comprise identifying historical training data used to train the ML model and a historical output of the ML model, indicating historical performance of the real-world system, determined by the ML model based on processing the historical training data identified. The computer-implemented method may further comprise determining whether there is at least one sustained change between the sensor data received and the historical training data identified based on a comparison between the current output of the ML model and the historical output of the ML model. The computer-implemented method may further comprise, responsive to determining that there is at least one sustained change, automatically detecting that the ML model should be retrained.
[0006] The computer-implemented method may further comprise retraining the ML model responsive to automatically detecting that the ML model should be retrained. The retraining may employ at least a portion of the sensor data received.
[0007] According to an embodiment of the computer-implemented method, a) the sensor data received represents time-series sensor data, b) the historical training data identified represents time-series historical training data, or c) any combination of a) and b).
[0008] In an example embodiment, a) the time series sensor data may be non-stationary multivariate time series sensor data, b) the time series historical training data may be non-stationary multivariate time series historical training data, or any combination of a) and b).
[0009] The identifying may include a) searching a look-up table (LUT) in a database, b) selecting the historical training data based on a least one training range, or c) any combination of a) and b).
[0010] The determining may include employing a statistical changepoint detection method. The statistical changepoint detection method may be insensitive to individual anomalies between the current output of the ML model and the historical output of the ML model.- 2 - 5083032. vlDocket No. 1086.2107001
[0011] The sensor data received may be associated with a plant process, an electrical power grid operation process, a hydrocarbon exploration process, a hydrocarbon production process, a mining operation process, a process of the real-world system, or any combination of processes of the real-world system for non-limiting examples.
[0012] The sensor data received may be industrial sensor data for non-limiting example. The real-world system may be an industrial, real-world system for non-limiting example.
[0013] According to example embodiments, a computer-implemented method for automating ML model selection may comprise receiving at least one criterion. The computer-implemented method may further comprise iteratively evaluating each candidate ML model of a plurality of candidate ML models until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model of the given iteration meets the at least one criterion and ii) evaluating the candidate ML model of an iteration subsequent to the given iteration exceeds an evaluation limit. Candidate ML models for evaluating (during the iteratively evaluating) may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be a selected optimum ML model.
[0014] The at least one criterion received may include a) at least one training range, b) a list of hyperparameters for a hyperparameter search space, c) at least one optimization metric, d) a time budget for the iteratively evaluating, or e) any combination of a)-d).
[0015] The computer-implemented method may further comprise monitoring performance of at least one asset in a real-world system using the selected optimum ML model.
[0016] The iteratively evaluating may include employing a) a BlendSearch method, b) a random search method, c) a Bayesian optimization method, d) a particle swarm optimization method, e) a genetic method, f) a grid search method, g) a dynamic search method, h) other search method, or i) any combination of a)-h).
[0017] The plurality of candidate ML models may include a) an unsupervised ML model, b) a supervised ML model, c) a regression ML model, d) a classification ML model, e) a clustering ML model, or f) any combination of a)-e).
[0018] The iteratively evaluating may include employing at least one metric. A metric of the at least one metric may include a classification metric, a regression metric, or a clustering metric for non-limiting examples.- 3 - 5083032. vlDocketNo. 1086.2107001
[0019] The iteratively evaluating may include employing a plurality of metrics with respective weights.
[0020] According to example embodiments, a computer-implemented method for automating ML model hyperparameter selection may comprise receiving at least one criterion and a hyperparameter search space. The computer-implemented method may further comprise iteratively evaluating hyperparameter values, within the hyperparameter search space received, for hyperparameters of a ML model until i) a result of the iteratively evaluating of a given iteration indicates that hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit. Hyperparameter values for evaluating may be selected based on learned performance of the ML model and learned computational cost of the evaluating. The hyperparameter values of the given iteration may be selected optimum hyperparameters for the ML model.
[0021] According to example embodiments, a computer-implemented method for automating ML model and ML model hyperparameter selection may comprise receiving at least one criterion and a hyperparameter search space. The computer-implemented method may further comprise iteratively evaluating each candidate ML model of a plurality of candidate ML models and hyperparameter values, within the hyperparameter search space received, for hyperparameters of each candidate ML model until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model and hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating the candidate ML model and hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit. Candidate ML models and hyperparameter values for evaluating may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be a selected optimum ML model and the hyperparameter values of the given iteration may be selected optimum hyperparameter values.
[0022] According to example embodiments, a computer-implemented method for automating ML model training data selection may comprise receiving a time-series dataset for monitoring performance of an asset of a real-world system. The computer-implemented method may further comprise producing a reduced time-series dataset by excluding a portion of data of the time-series dataset received. The computer-implemented method may further comprise iteratively processing the reduced time-series dataset to identify at least one data- 4 - 5083032. vlDocket No. 1086.2107001distribution in the reduced time-series dataset. The computer-implemented method may further comprise automatically selecting a given data distribution from the at least one data distribution identified, wherein the given data distribution selected is selected as training data for a ML model and the given data distribution is selected automatically based on a temporal correspondence between the given data distribution selected and a target timeframe for monitoring performance of the asset with the ML model.
[0023] Selecting the given data distribution may include identifying a target behavior in the given data distribution and automatically selecting the given data distribution based on a) the temporal correspondence between the given data distribution selected and the target timeframe and b) the identifying the target behavior in the given data distribution.
[0024] The target behavior may include a) at least one anomaly, b) at least one failure signature, or c) any combination of a) and b).
[0025] The computer-implemented method may further comprise training the ML model using the given data distribution selected automatically. The training may produce a trained ML model. The computer-implemented method may further comprise deploying the trained ML model in the target timeframe.
[0026] The portion of data excluded may correspond to a) a time period for which the asset was known to be offline, b) a time period for which the asset was associated with a confirmed failure event, or c) any combination of a) and b).
[0027] The time-series dataset received may, for non-limiting example, be sensor data received from at least one sensor coupled to the asset.
[0028] The time-series dataset may include non- stationary multivariate time series data or stationary multivariate time series data for non-limiting examples.
[0029] The given data distribution may be selected based on the temporal correspondence and a determination that the given data distribution, in its entirety, encompasses a training data portion and a test data portion in a single distribution.
[0030] The iteratively processing may include determining whether the reduced timeseries dataset, in its entirety, corresponds to one distribution. Responsive to determining that the reduced time-series dataset, in its entirety, does not correspond to one distribution, the iteratively processing may further include: iteratively i) dividing the reduced time-series dataset into a plurality of datasets, the iteratively dividing including dividing each dataset of the plurality of datasets into a candidate training data portion and test data portion and ii) evaluating whether each dataset of the plurality of datasets corresponds to one distribution,- 5 - 5083032. vlDocketNo. 1086.2107001until a) each dataset of the plurality of datasets corresponds to a respective one distribution or b) a total number of maximum iterations has been performed. The at least one data distribution identified may be composed of each candidate training data portion.
[0031] According to an example embodiment, a computer-based system for automating ML model retraining detection may comprise at least one processor and at least one memory with computer code instructions stored thereon. The at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system to receive sensor data from at least one sensor coupled to a real-world system. The sensor data may be employed by a ML model trained to monitor performance of the real-world system. A current output of the ML model, determined by the ML model based on processing the sensor data received, may indicate current performance of the real-world system. The at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system to identify historical training data used to train the ML model and a historical output of the ML model. The historical output may indicate historical performance of the real-world system, determined by the ML model based on processing the historical training data identified. The at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system to determine whether there is at least one sustained change between the sensor data received and the historical training data identified based on a comparison between the current output of the ML model and the historical output of the ML model. Responsive to determining that there is at least one sustained change, the at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system to automatically detect that the ML model should be retrained.
[0032] Alternative computer-based system embodiments for automating ML model retraining detection parallel those described above in connection with the example computer-implemented method embodiments for automating ML model retraining detection.
[0033] According to example embodiments, a computer-based system for automating ML model selection may comprise at least one processor and at least one memory with computer code instructions stored thereon. The at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system to receive at least one criterion and iteratively evaluate each candidate ML model of a plurality of candidate ML models until: i) a result of the iteratively evaluating of a given iteration- 6 - 5083032. vlDocketNo. 1086.2107001indicates that the candidate ML model of the given iteration meets the at least one criterion and ii) evaluating the candidate ML model of an iteration subsequent to the given iteration exceeds an evaluation limit. Candidate ML models for evaluating may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be a selected optimum ML model.
[0034] Alternative computer-based system embodiments for automating ML model selection parallel those described above in connection with the example computer-implemented method embodiments for automating ML model selection.
[0035] According to example embodiments, a computer-based system for automating ML model hyperparameter selection may comprise at least one processor and at least one memory with computer code instructions stored thereon. The at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system to receive at least one criterion and a hyperparameter search space. The at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system to iteratively evaluate hyperparameter values, within the hyperparameter search space received, for hyperparameters of a ML model until i) a result of the iteratively evaluating of a given iteration indicates that hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit.Hyperparameter values for evaluating may be selected based on learned performance of the ML model and learned computational cost of the evaluating. The hyperparameter values of the given iteration may be selected optimum hyperparameters for the ML model.
[0036] According to example embodiments, a computer-based system for automating ML model and ML model hyperparameter selection may comprise at least one processor and at least one memory with computer code instructions stored thereon. The at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system to receive at least one criterion and a hyperparameter search space. The at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system to iteratively evaluate each candidate ML model of a plurality of candidate ML models and hyperparameter values, within the hyperparameter search space received, for hyperparameters of each candidate ML model until: i) a result of the iteratively evaluating of- 7 - 5083032. vlDocket No. 1086.2107001a given iteration indicates that the candidate ML model and hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating the candidate ML model and hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit. Candidate ML models and hyperparameter values for evaluating may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be a selected optimum ML model and the hyperparameter values of the given iteration may be selected optimum hyperparameter values.
[0037] According to example embodiments, a computer-based system for automating ML model training data selection may comprise at least one processor and at least one memory with computer code instructions stored thereon. The at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system to receive a time-series dataset for monitoring performance of an asset of a real-world system, produce a reduced time-series dataset by excluding a portion of data of the time-series dataset received, iteratively process the reduced time-series dataset to identify at least one data distribution in the reduced time-series dataset, and automatically select a given data distribution from the at least one data distribution identified. The given data distribution selected may be selected as training data for a ML model and the given data distribution may be selected automatically based on a temporal correspondence between the given data distribution selected and a target timeframe for monitoring performance of the asset with the ML model.
[0038] Alternative computer-based system embodiments for automating ML model training data selection parallel those described above in connection with the example computer-implemented method embodiments for automating ML model training data selection.
[0039] According to example embodiments, a non-transitory computer-readable medium for automating ML model retraining detection may have encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to receive sensor data from at least one sensor coupled to a real-world system. The sensor data may be employed by a ML model trained to monitor performance of the real-world system. A current output of the ML model, determined by the ML model based on processing the sensor data received, may indicate current performance of the real-world system. The at least one processor may be further caused to identify historical training data- 8 - 5083032. vlDocketNo. 1086.2107001used to train the ML model and a historical output of the ML model, indicating historical performance of the real-world system, determined by the ML model based on processing the historical training data identified. The at least one processor may be further caused to determine whether there is at least one sustained change between the sensor data received and the historical training data identified based on a comparison between the current output of the ML model and the historical output of the ML model. Responsive to determining that there is at least one sustained change, the at least one processor may be further caused to automatically detect that the ML model should be retrained.
[0040] Alternative non-transitory computer-readable medium embodiments for automating ML model retraining detection parallel those described above in connection with the example computer-implemented method embodiments for automating ML model retraining detection.
[0041] According to example embodiments, a non-transitory computer-readable medium for automating ML model selection may have encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to receive at least one criterion and to iteratively evaluate each candidate ML model of a plurality of candidate ML models until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model of the given iteration meets the at least one criterion and ii) evaluating the candidate ML model of an iteration subsequent to the given iteration exceeds an evaluation limit. Candidate ML models for evaluating may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be a selected optimum ML model.
[0042] Alternative non-transitory computer-readable medium embodiments for automating ML model selection parallel those described above in connection with the example computer-implemented method embodiments for automating ML model selection.
[0043] According to example embodiments, a non-transitory computer-readable medium for automating ML model hyperparameter selection may have encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to receive at least one criterion and a hyperparameter search space. The at least one processor may be further caused to iteratively evaluate hyperparameter values, within the hyperparameter search space received, for hyperparameters of a ML model until i) a result of the iteratively evaluating of a given iteration indicates that hyperparameter values of the- 9 - 5083032. vlDocket No. 1086.2107001given iteration meet the at least one criterion and ii) evaluating hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit, wherein hyperparameter values for evaluating are selected based on learned performance of the ML model and learned computational cost of the evaluating. The hyperparameter values of the given iteration may be selected optimum hyperparameters for the ML model.
[0044] According to example embodiments, a non-transitory computer-readable medium for automating ML model and ML model hyperparameter selection may have encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to receive at least one criterion and a hyperparameter search space and to iteratively evaluate each candidate ML model of a plurality of candidate ML models and hyperparameter values, within the hyperparameter search space received, for hyperparameters of each candidate ML model until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model and hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating the candidate ML model and hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit. Candidate ML models and hyperparameter values for evaluating may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be a selected optimum ML model and the hyperparameter values of the given iteration may be selected optimum hyperparameter values.
[0045] According to example embodiments, a non-transitory computer-readable medium for automating ML model training data selection may have encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to receive a time-series dataset for monitoring performance of an asset of a real-world system. The at least one processor may be further caused to produce a reduced time-series dataset by excluding a portion of data of the time-series dataset received, iteratively process the reduced time-series dataset to identify at least one data distribution in the reduced time-series dataset, and automatically select a given data distribution from the at least one data distribution identified. The given data distribution selected may be selected as training data for a ML model and the given data distribution may be selected automatically based on a temporal correspondence between the given data distribution selected and a target timeframe for monitoring performance of the asset with the ML model.- 10 - 5083032. vlDocket No. 1086.2107001
[0046] Alternative non-transitory computer-readable medium embodiments for automating ML model training data selection parallel those described above in connection with the example computer-implemented method embodiments for automating ML model selection.
[0047] It should be understood that example embodiments disclosed herein can be implemented in any combination and in the form of a method, apparatus, system, or non-transitory computer readable medium with program codes embodied thereon.BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The foregoing will be apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments.
[0049] FIG. 1 A is a block diagram of an example embodiment of a computing environment in which a user is using a computer-based system for automating machinelearning (ML) model retraining detection.
[0050] FIG. IB is a block diagram of an example embodiment of the computing environment of FIG. 1 A in which the user is using the computer-based system for automating ML model selection.
[0051] FIG. 1C is a block diagram of an example embodiment of the computing environment of FIG. 1 A in which the user is using the computer-based system for automating ML model hyperparameter selection.
[0052] FIG. ID is a block diagram of an example embodiment of the computing environment of FIG. 1 A in which the user is using the computer-based system for automating ML model and ML model hyperparameter selection.
[0053] FIG. IE is a block diagram of an example embodiment of the computing environment of FIG. 1 A in which the user is using the computer-based system for automating ML model training data selection.
[0054] FIG. 2 is a flow diagram of an example embodiment of a workflow for automated retraining of artificial intelligence (Al) methods disclosed herein.
[0055] FIG. 3 is a plot of an example embodiment of sensor data from four sensors in different positions at a same piece of equipment, centered and scaled.- 11 - 5083032. vlDocket No. 1086.2107001
[0056] FIG. 4 is a plot of an example embodiment of a univariate signal derived from data of the plot of FIG. 3.
[0057] FIG. 5 is a graph of an example embodiment of multivariate time series data from equipment sensors.
[0058] FIG. 6 is a graph of an example embodiment of anomaly score values.
[0059] FIG. 7 is another graph of an example embodiment of anomaly score values.
[0060] FIG. 8 is a flow diagram of an example embodiment of a workflow for automated training data selection.
[0061] FIG. 9 is a flow diagram of an example embodiment of a workflow for finding data distributions in time series sensor data.
[0062] FIG. 10 is a flow diagram of an example embodiment of a workflow for automatic (auto) ML.
[0063] FIG. 11 A is a plot of an example embodiment of anomaly scores generated by a ML model tuned by a user.
[0064] FIG. 1 IB is a plot of an example embodiment of anomaly scores generated by a ML model created by an A assisted workflow.
[0065] FIG. 12 is another plot of an example embodiment of a univariate signal.
[0066] FIG. 13 is a flow diagram of an example embodiment of a computer-implemented method for automating ML model retraining detection.
[0067] FIG. 14 is a flow diagram of an example embodiment of a computer-implemented method for automating ML model selection.
[0068] FIG. 15 is a flow diagram of an example embodiment of a computer-implemented method for automating ML model hyperparameter selection.
[0069] FIG. 16 is a flow diagram of an example embodiment of a computer-implemented method for automating ML model and ML model hyperparameter selection.
[0070] FIG. 17 is a flow diagram of an example embodiment of a computer-implemented method for automating ML model training data selection.
[0071] FIG. 18 is a block diagram of an example of an internal structure of a computer in which various embodiments of the present disclosure may be implemented.DETAILED DESCRIPTION
[0072] A description of example embodiments follows.- 12 - 5083032. vlDocket No. 1086.2107001
[0073] A “workflow” as referenced herein may be referred to interchangeably as a pipeline or method.
[0074] In the industrial internet-of-things (IIoT), data are recorded and shared from a vast number of sensors coupled to many different kinds of pieces of equipment, referred to as assets, which may or may not be physically connected. A wealth of data generated by such sensors creates a need for data processing as it is impossible for operators to watch and analyze all sensors, e.g., on a continuous basis. The information contained in the data cannot be processed and summarized by an operator in real-time, much less in a way that is actionable. An example embodiment disclosed herein may deploy a ML model, not only to summarize such information, but also to detect situations that deviate from normal ( / .<?., “the norm”) and, therefore, may prompt intervention.
[0075] Examples of such deviations exist across different industries. As a first nonlimiting example, a vibration sensor for a mechanical bearing may be considered. Mechanical bearings are very common and can be found in industrial manufacturing equipment, in water and wind turbines, in gearboxes for trains and other modes of transportation, and in drilling equipment for the energy and mining industries, to name a few non-limiting examples.Vibrations recorded from such bearings are typically a stationary time series. That time series can be multivariate, as there may be multiple vibration sensors attached to the same bearing. Moreover, operators may have hundreds of (similar) bearings to monitor, as may be the case in, for non-limiting example, a wind farm. In practice, operators will not want to monitor each of these time series and will want to be alerted only in case a certain type of action is needed.
[0076] For instance, and for non-limiting examples, an operator may want to know if a certain sensor generates vibrations higher than usual, or if a certain set of sensors that are typically correlated, decouple. In other words, it is useful for an operator to be alerted about disruptions and anomalies. Going one step beyond anomaly detection, preventative action may be recommended when patterns start to deviate from their usual shapes at an early stage, which can thereby pre-empt a disruption.
[0077] Typically, the identification of patterns, such as trends or anomalies in univariate or multivariate sensor data, is accomplished through statistical or ML models. Such models are usually trained from data, then deployed in a live environment (a real-world environment) where the models make assessments relative to the modus operand! reflected in the data at the time when a ML model was trained.- 13 - 5083032. vlDocket No. 1086.2107001
[0078] Statistical or ML models are trained from a certain data basis and predictions by the ML models or assessments based on the predictions can vary widely, depending on which data were selected to train the ML models. Moreover, some ML models can yield very different results, depending on which hyperparameters were selected for the ML models. In practice, hyperparameter selection is a data driven process, so the hyperparameter selection process also depends upon the selection of the training data. An example embodiment disclosed herein enables both training data and a corresponding set of hyperparameters to be selected to a certain degree of optimality, which makes those aspects of training a ML model for asset performance monitoring a reproducible process that does not rely on a user to make potentially subjective decisions.
[0079] For non-limiting example, with focus on training and deployment of ML models for asset performance monitoring, many practical deployments of such ML models may collect data, train models based on the data collected, and then deploy the ML models to make assessments in real time. However, the implicit assumption in that order of proceeding is that the original data used to train the ML models continues to be a representative sample of current, real-world operations.
[0080] In practice, reality can drift (deviate) away from the state of affairs at a time of training. Non-limiting examples of reasons why sensor data values can start diverging (deviating) from values at the time of training are: wear, gradual degradation of a lubricant used (or a different selection of lubricant), fouling, catalyst deactivation, or more mundane reasons, such as different weather conditions, to name a few non-limiting examples.
[0081] When data drifts from a baseline at a time of training, ML models that rely on those data need to be retrained. A conventional approach is to rely on an expert to notice that a ML model is not performing as expected, to identify data drift as the cause, and then for the expert to select modified training data that represents a new / current state of affairs, and to retrain the ML model based on the selected modified training data. Success in such a scenario depends on the expert correctly executing each of such actions and, even if so, manual intervention and time investment from the expert is relied on and such actions cannot be performed in real time.
[0082] Example embodiments disclosed herein may automatically train and retrain an ML model, the latter, e.g., whenever data drift from a baseline. Example embodiments disclosed herein may: (i) automate selection of an optimal set of training data (original or new) to (re)train the ML model, (ii) automate selection of a best (optimum) ML model from a- 14 - 5083032. vlDocket No. 1086.2107001set of candidate ML models, and (iii) automate detection of a need to retrain a ML model. If the latter is positive, (i) and / or (ii) can be repeated. According to an example embodiment, by the time there is a need to retrain a ML model, more recent data may be included in a new training set for retraining the ML model, such as the ML model 112 of FIG. 1 A, disclosed below.
[0083] FIG. 1 A is a block diagram of an example embodiment of a computing environment 100 in which a user 102, such as an operator for non-limiting example, is using a computer-based system 104 that automates machine-learning (ML) model retraining detection. The computer-based system 104 may comprise at least one processor (not shown) and at least one memory (not shown) with computer code instructions (not shown) stored thereon, such as the central processor unit 1817 coupled to the memory 1807 that includes the computer code instructions 1809 stored thereon, disclosed further below with regard to FIG.18 for non-limiting example.
[0084] Continuing with reference to FIG. 1 A, according to an example embodiment, the computer-based system 104 may be used for automating ML model retraining detection, z.e., the need to retrain a ML model. The at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system 104 to receive sensor data 106 from at least one sensor 108 coupled to a real -world system 110. The sensor data 106 may be employed by a ML model 112 trained to monitor performance of the real -world system 110.
[0085] The sensor data 106 received may be associated with a plant process, an electrical power grid operation process, a hydrocarbon exploration process, a hydrocarbon production process, a mining operation process, a process of the real -world system 110, or any processes or combination of processes of the real -world system 110 for non-limiting examples. The sensor data 106 received may be industrial sensor data for non-limiting example. The real-world system 110 may be an industrial, real -world system for non-limiting example.
[0086] A current output (not shown) of the ML model 112, determined by the ML model 112 based on processing the sensor data 106 received, may indicate current performance of the real -world system 110. The at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system 104 to identify historical training data (not shown) used to train the ML model 112 and a historical output (not shown) of the ML model 112.- 15 - 5083032. vlDocket No. 1086.2107001
[0087] The historical output may indicate historical performance of the real-world system 110, determined by the ML model 112 based on processing the historical training data identified. The at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system 104 to determine whether there is at least one sustained change (not shown) between the sensor data 106 received and the historical training data identified. The at least one sustained change may be based on a comparison between the current output of the ML model 112 and the historical output of the ML model 112. Responsive to determining that there is at least one sustained change, the at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system 104 to automatically detect that the ML model 112 should be retrained.
[0088] The at least one sustained change may be asset specific and also customer specific for non-limiting example. A sustained change of the at least one sustained change may represent at least two data points in order to be considered a sustained change for nonlimiting example. According to a non-limiting example embodiment, at least 2% (or other %) of the training data points relative to a total number of training data points used to train the training model may be a threshold number of data points considered to represent a sustained change.
[0089] As such, the user 102 need not be an expert who monitors output (not shown) from the ML model 112 to determine that ML model 112 should be retrained. Indeed, the user 102, even though an expert, may not notice that the ML model 112 is not performing as expected. The computer-based system 104, however, automatically detects that the ML model 112 should be retrained, ensuring success and obviating manual intervention and time investment from the user 102.
[0090] Alternative computer-based system 104 embodiments for automating ML model retraining detection parallel those described in connection with example computer-implemented method embodiments of a computer-implemented method (1300) for automating ML model retraining detection, disclosed further below with regard to FIG. 13.
[0091] Continuing with reference to FIG. 1 A, according to example embodiments, the computer-based system 104 may be used for automating ML model selection, such as disclosed below with regard to FIG. IB.
[0092] FIG. IB is a block diagram of an example embodiment of the computing environment 100 of FIG. 1A. In the example embodiment of FIG. IB, the user 102 is using- 16 - 5083032. vlDocket No. 1086.2107001the computer-based system 104 for automating ML model selection. With reference to FIG. IB, the at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system 104 to receive at least one criterion 114 and, optionally, receive an evaluation limit 118. If, for instance, the evaluation limit 118 is not received, the computer-based system 104 may perform functionality utilizing a pre-set evaluation limit 118. To continue, the at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system 104 to iteratively evaluate each candidate ML model of a plurality of candidate ML models 116 until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model (not shown) of the given iteration meets the at least one criterion 114 and ii) evaluating the candidate ML model of an iteration subsequent to the given iteration exceeds the evaluation limit 118. Candidate ML models for evaluating may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be a selected optimum ML model 120.
[0093] Alternative computer-based system 104 embodiments for automating ML model selection parallel those described in connection with example computer-implemented method embodiments of a computer-implemented method (1400) for automating ML model selection, disclosed further below with regard to FIG. 14.
[0094] Continuing with reference to FIG. IB, according to example embodiments, the computer-based system 104 may be used for automating ML model selection, such as disclosed below with regard to FIG. 1C.
[0095] FIG. 1C is a block diagram of an example embodiment of the computing environment of FIG. 1 A. In the example embodiment of FIG. 1C, the user 102 is using the computer-based system 104 for automating ML model hyperparameter selection. With reference to FIG. 1C, the at least one processor and the at least one memory, with the computer code instructions, may be configured to cause computer-based system 104 to receive at least one criterion 144, a hyperparameter search space 122, and, optionally, an evaluation limit 148. If, for instance, the evaluation limit 148 is not received, the computer-based system 104 may perform functionality utilizing a pre-set evaluation limit 148. The at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system 104 to iteratively evaluate hyperparameter values (not shown), within the hyperparameter search space 122 received, for- 17 - 5083032. vlDocket No. 1086.2107001hyperparameters of the ML model 112 until i) a result (not shown) of the iteratively evaluating of a given iteration indicates that hyperparameter values (not shown) of the given iteration meet the at least one criterion 144 and ii) evaluating hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit 148. Hyperparameter values for evaluating may be selected based on learned performance of the ML model 112 and learned computational cost of the evaluating. The hyperparameter values of the given iteration may be selected optimum hyperparameters 124 for the ML model 112.
[0096] Alternative computer-based system 104 embodiments for automating ML model hyperparameter selection parallel those described in connection with example computer-implemented method embodiments of a computer-implemented method (1500) for automating ML model hyperparameter selection as disclosed further below with regard to FIG. 15.
[0097] According to example embodiments, the computer-based system 104 may be used for automating ML model and ML model hyperparameter selection, such as disclosed below with regard to FIG. ID.
[0098] FIG. ID is a block diagram of an example embodiment of the computing environment 100 of FIG. 1A. In the example embodiment of FIG. ID, the user 102 is using the computer-based system 104 for automating ML model and ML model hyperparameter selection. The at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system 104 to receive at least one criterion 154 and a hyperparameter search space 152. The at least one processor and the at least one memory, with the computer code instructions, may be further configured to cause the computer-based system 104 to iteratively evaluate each candidate ML model of the plurality of candidate ML models 116 and hyperparameter values (not shown), within the hyperparameter search space 152 received, for hyperparameters of each candidate ML model until: i) a result (not shown) of the iteratively evaluating of a given iteration indicates that the candidate ML model and hyperparameter values of the given iteration meet the at least one criterion 154 and ii) evaluating the candidate ML model and hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit (not shown). Candidate ML models and hyperparameter values for evaluating may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be the selected optimum ML model 121- 18 - 5083032. vlDocket No. 1086.2107001and the hyperparameter values of the given iteration may be selected optimum hyperparameter values 126.
[0099] Alternative computer-based system 104 embodiments for automating ML model and ML model hyperparameter selection parallel those described in connection with example computer-implemented method embodiments of a computer-implemented method (1600) for automating ML model and ML model hyperparameter selection as disclosed further below with regard to FIG. 16.
[0100] According to example embodiments, the computer-based system 104 may be used for automating ML model training data selection, such as disclosed below with regard to FIG. IE.
[0101] FIG. IE is a block diagram of an example embodiment of the computing environment of FIG. 1 A. In the example embodiment of FIG. IE, the user 102 is using the computer-based system 104 for automating ML model training data selection. With reference to FIG. IE, the at least one processor and the at least one memory, with the computer code instructions, may be configured to cause the computer-based system 104 to receive a timeseries dataset 128 for monitoring performance of an asset 130 of the real -world system 110, produce a reduced time-series dataset (not shown) by excluding a portion of data (not shown) of the time-series dataset 128 received, iteratively process the reduced time-series dataset to identify at least one data distribution (not shown) in the reduced time-series dataset, and automatically select a given data distribution 134 from the at least one data distribution identified. The given data distribution 134 selected may be selected as training data for the ML model 112 and the given data distribution 134 may be selected automatically based on a temporal correspondence between the given data distribution 134 selected and a target timeframe 132 for monitoring performance of the asset 130 with the ML model 112.
[0102] Alternative computer-based system 104 embodiments for automating ML model training data selection parallel those described in connection with example computer-implemented method embodiments of a computer-implemented method (1700) for automating ML model training data selection as disclosed further below with regard to FIG.17.
[0103] Further technical details are disclosed below.Workflow Overview- 19 - 5083032. vlDocket No. 1086.2107001
[0104] An example embodiment disclosed herein may constitute a cyclical artificial intelligence (Al) workflow that ensures that ML models can be constructed that best represent real-world operations at any point in time, which includes detection (including automatic detection) of a need to retrain ML models, such that the ML models stay up to date with changes in the real-world system the ML models monitor.
[0105] The Al workflow disclosed herein includes multiple actions. In an embodiment, at first, sensor data may be collected to train a ML model in an initial model training phase. The initial model training phase benefits from example embodiments disclosed herein, such as: automated selection of a correct range(s) in an initial set of training data and automated selection of an optimal ML model (both type of model and hyperparameters).
[0106] Once an initial ML model has been built, the ML model may be deployed and connected to a data feed, e.g., a continuous data feed, from operations, e.g., asset(s) from a real-world system. At all points in time, new incoming data may be stored along with the preexisting data, for example, in a database for non-limiting example. The ML model may also be referred to as an asset performance monitoring (APM) agent and may proceed to make predictions from the live (real-time) data. Those predictions may, in turn, be transformed and subjected to at least one evaluation criterion, to determine if a current state of operations deviates from assumptions. If so, a system that employs the APM agent may produce an alert.
[0107] An example embodiment of a retraining detection agent may evaluate if the operating conditions still align with those that a latest version of a ML model(s) had been trained from. If not, the retraining detection agent may initiate training of a new ML model based on the latest, most complete set of data. Such a complete set of data may include both original training data and any new data that have come in from the system since deployment of the ML model. The system may either initiate ML model retraining, automatically, or alert an operator to do so. Some reasons for choosing automatic vs. operator intervention include a level of customer confidence in Al, as well as possible regulatory aspects that require operator intervention for non-limiting examples. An example embodiment of an Al workflow is disclosed below with regard to FIG. 2.
[0108] FIG. 2 is a flow diagram of an example embodiment of a workflow 200 for automated retraining of artificial intelligence (Al) methods, such as a ML model disclosed herein. Continuing with reference to FIG. 2, the workflow 200 may utilize original training data 241, once, along with new sensor data 242, received continuously, to produce all data available 243. Based on all data available 243, an automated selection of training ranges 244- 20 - 5083032. vlDocket No. 1086.2107001may be performed, as needed, followed by automated ML model selection and training 245, as needed. The new sensor data 242 may be employed, continuously, for live agent prediction 246 and retraining detection 247, continuously. A check 248 for data drift may be performed, resulting in automated selection of training ranges 244, as needed. An output 249 from the live agent prediction 246 may be checked 251 to determine if the output 249 exceeds a limit. If so, an alert 253 may be generated and, otherwise, a decision 255 may be made not to generate the alert 253.Practical implementation of asset performance monitoring methods
[0109] Methods for APM themselves can be highly complex. However, the results from such methods should be both easy to interpret and duly actionable. Therefore, successful applications of such methods typically summarize the methods’ findings into one or just a few signals to monitor. Such signals are a low dimensional representation of higher dimensional sensor data. The low dimensional representation can be computed in various ways, for instance, by computing distances to (moving) averages or to clusters of data points, by computing latent variables that represent a ‘normal’ mode of operations through some form of dimension reduction, such as principal component analysis, partial least squares, or an autoencoder, or as low-dimensional predictions from a regression model for non-limiting examples.
[0110] To make a low dimensional representation actionable, typically the low dimensional representation itself is transformed into a probability, which then represents how likely it is to encounter a given sensor reading given what has been identified as a ‘normal’ mode of operations. The low dimensional representation can also be subjected to a statistical hypothesis test that assesses how likely a given sensor reading is, given an estimated normal state of operations. An example of translation (transformation) of sensor signals is plotted in FIG. 3 and FIG. 4, disclosed below.
[0111] FIG. 3 is a plot 300 of a sensor data signal 333 plotted over time 331 from four sensors coupled to a same piece of equipment (not shown) at different locations. For graphical purposes, raw data output from the four sensors was centered and scaled prior to plotting. A transformation of these data is disclosed below with regard to FIG. 4, in which such data are transformed into a signal that quantifies how anomalous an individual data point is. Anomaly detection is a common objective in asset performance monitoring, but other objectives can exist as well, such as prediction or forecasting for non-limiting- 21 - 5083032. vlDocket No. 1086.2107001examples. For non-limiting example, a reading in excess of 0.5 (50%) may be considered an ‘anomaly,’ which can then automatically produce an alert to be investigated and acknowledged by an operator, such as the user 102 of FIGS. 1 A-E, disclosed above.
[0112] FIG. 4 is a plot 400 of an example embodiment of a univariate signal 433 over time 431, derived (by a trained ML model) from data of the plot 300 of FIG. 3, disclosed above. Continuing with reference to FIG. 4, the plot 400 assesses how likely an individual point is anomalous. Readings above 0.5 are considered anomalous and are visualized as vertical, solid black lines.
[0113] In the plot 400 of FIG. 4, anomalous data points are accentuated by vertical, solid black lines. As can be interpreted from the plot 400 of FIG. 4, very few anomalies are detected for an extended period of time, which indicates that the piece of equipment mostly ran in a mode of operations in accordance with a ‘normal’ established at a time of training a ML model. After a piece (component) of equipment was restarted in mid-2022, however, the mode of operations deviated from a state used to train the ML model. Thenceforth, the model (ML model used to process the data of the plot 300 of FIG. 3 to generate the data of the plot 400 of FIG. 4) labels the majority of data points as ‘anomalous,’ whereas, in reality, such data points merely represent a change in the normal, which at this point should be accounted for by retraining the model.Automated selection of training data
[0114] An example embodiment of a methodology disclosed herein has an objective to automatically select training data, save time for a user, and simplify a challenging task of visualizing high-dimensional data and deciding correct training and test splits. Selection of training data is useful for supervised and semi-supervised ML models to properly learn a correct mapping between inputs and expected outputs. For semi-supervised anomaly detection models, training data may be used as a reference of what is normal, and deviations from that reference produce respective anomaly scores. Such is not true for supervised ML models.
[0115] A supervised model will use examples of normal and events to detect. However, for semi-supervised, only examples of normal are provided. Additionally, a supervised model is provided a failure / event to learn. As it is in supervised learning, such a ML model will not learn to detect anomalies. The ML model instead will learn to classify if input data is similar to a failure represented in the training data. Many anomalies can, however, occur, and a- 22 - 5083032. vlDocket No. 1086.2107001machine can fail or have anomalies that were not present in prior input data, as such anomalies may not occur until they happen.
[0116] Consequently, selecting better data for training creates a better anomaly detection model. Nonetheless, there are some challenges in selecting training data from sensors located on multiple parts (components) of equipment. Sensor data may represent non- stationary multivariate time-series data, with changing statistical properties over time, as illustrated in FIG. 5, disclosed below.
[0117] FIG. 5 is a graph 500 of multivariate time series data from equipment sensors where a y-axis 561 represents values of sensor data, and an x-axis 563 represents date and time of a measurement of such sensor data. In the graph 500, a first time series T1 data has a trend change 564 that increases with time. A second time series data T2 has a regime change 565 that includes two operation regimes over an entire time of observation. A third time series T3 data represents temperature sensor data from a temperature sensor, and the third time series T3 data has seasonality 566 corresponding to fluctuations of temperature during seasons (winter, spring, summer, fall).
[0118] As such, equipment may have multiple operating modes to adapt to a demand or product and may simultaneously be affected by an ambient temperature that is cold in winters and hot in summers for non-limiting examples. This non- stationary behavior poses a challenge for selecting representative ranges for training and test data for anomaly detection models. Therefore, it is useful to select training data that captures, e.g., all, existing variations, data distributions, and considers temporal dependency, such as seasonality.
[0119] It is common practice in ML to split time series data into training and test sets, which is known as train test split. The train (training) set is used to train a ML model, and the test set is used to validate performance of the ML model to ensure the ML model is not underfitted or overfitted. One of the most common train test splits for time series data is to use the 70% of the data for the training set and the remaining 30% of the data for the test set, starting from oldest to most recent data. This practice helps to preserve the most recent data for test and ensure good performance when the ML model analyzes new incoming data. However, this approach does not perform well on industrial assets because such equipment can operate differently during the last 30% and violates an assumption that train and test data belong to the same distribution. FIG. 6 and FIG. 7, disclosed below, show a transformation of data into a signal that quantifies how anomalous each individual data point of the data is via a representation of anomaly scores.- 23 - 5083032. vlDocket No. 1086.2107001
[0120] FIG. 6 is a graph 600 of an example embodiment of anomaly score values 662. In the graph 600, the x-axis represents the time 631 in dates and the y-axis represents the anomaly score values 662. The regions 635a-f are data excluded due to equipment being offline. The region 638 is where a confirmed equipment failure occurred. A region 639 includes data used for training. Finally, a line 637 in a middle location of the graph 600 indicates a threshold to distinguish between normal and anomalous data. FIG. 6 illustrates an issue where most of the data that was not selected for training has a significant increase in the anomaly scores. Therefore, based on FIG. 7, disclosed below, where the same experiment is repeated, using the same ML model used for FIG. 6, and only modifying the training data, a conclusion may be drawn that the data from training and test has different data distributions that were not captured with the common 70 / 30 split.
[0121] FIG. 7 is another graph 700 of an example embodiment of anomaly score values 762. In the graph 700, the x-axis represents time 731 via dates and the y-axis represents anomaly score values 762. Regions 735a-f include data excluded due to equipment being offline. A region 738 represents a time in which a confirmed equipment failure occurred. Regions 739a-d include data used for training. Finally, a line 737, in a middle of the graph 700, indicates a threshold to distinguish between normal and anomalous data.
[0122] To select more representative training data, an example embodiment disclosed herein identifies diverse data distributions and takes data representing the oldest part of each distribution, leaving data representing a most recent part of a time series data set for testing. Following this approach ensures that diverse sensors’ behaviours are captured in training data, and ensures that data left for test is close to a present time to guarantee correct behaviour when a ML model is used for analyzing new incoming data at a present time. In FIG. 7, the experiment in FIG. 6 is repeated only changing the training data, using an example embodiment disclosed herein, e.g., method 1700 of FIG. 17. As a consequence, the ML model achieves a higher performance than the ML model trained with conventional traintest split data and ensures the existance of test data for the different data distributions.Selection of the training data is accomplished using, e.g., an example embodiment of a methodology (workflow) disclosed below with regard to FIG. 8.
[0123] FIG. 8 is a flow diagram of an example embodiment of a workflow 800 for automated training data selection. The workflow 800 starts at 802, and a first action is to take as inputs the sensor data 806, offline regions 835, and confirmed failures / events 807 and perform a train test split 808 excluding offline regions 835 and confirmed failures / events 807.- 24 - 5083032. vlDocket No. 1086.2107001Such a split may be a 70 / 30 split or other split, such as 69% train, 31% test, or other train / test ratio for non-limiting examples. Such confirmed failures / events 807 may be confirmed by human intervention or another method. It is common for industrial systems (e.g., Enterprise Asset Management / EAM) to have data of an asset of a machine be recorded, as well as different events associated with the machine. Engineers may record any maintenance, observation, failure, or repair that they perform on a daily basis for non-limiting examples. An example embodiment may leverage such recorded information for determining training ranges.
[0124] Continuing with reference to FIG. 8, the workflow 800 may check 810 if the whole (entire) data belongs to a single distribution. To validate that the sensor data 806 belongs to a single distribution, multiple methods can be applied. For instance, univariate data distribution tests, multivariate distribution tests, temporal structure evaluation and / or others may be employed for non-limiting examples. While any method can be used; however, the results presented are based on use of a Gaussian Mixture Model (GMM) model to verify a single distribution in the data. A GMM model was chosen to ensure that validation considers multivariate time series and relationships between sensors (D. Ververidis and C. Kotropoulos, (2008). "Gaussian Mixture Modeling by Exploiting the Mahalanobis Distance," in IEEE Transactions on Signal Processing, vol. 56, no. 7, 2797-2811). For non-limiting example, a GMM may be the best option based on the experiments, because it considers multi time series, the interaction between sensors, and is fast enough to iterate. In contrast, the other tested methods didn’t perform as well as GMM. However, the workflow 800 could still give good results using other methods.
[0125] Continuing with reference to FIG. 8, if the check 810 determines that the whole data belongs to a single distribution, the search for distributions finishes 823 in the example embodiment. On the other hand, if the data contains different distributions, the search continues, in order to capture a better representation. To continue with the search, a linear trend is added as an additional time series into all the time series of the sensor data 806 to promote time dependency, wherein samples near in time should be closer in distance than ones farther in time.
[0126] Such an additional linear trend is a diagonal line that increases with time, gradually. Because the line increases gradually, the points of the line that are nearby are closer than the points that are far away. For example, if one takes a starting point and ending point, such points are not touching (farther in distance). However, by choosing points that are- 25 - 5083032. vlDocket No. 1086.2107001next to each other, they are touching (close in distance). Therefore, in clustering, a method may group points together based on how far apart they are. Adding this trend will allow the sensor readings that are recoded one after another to be considered closer than respective values of a first recording and last recording.
[0127] Next, the data distributions are found 812 and isolated to perform the train test split 814 as before, for each discovered distribution. Afterwards, each discovered distribution is tested 816 to check if it represents a single (sole) distribution. If the check 818 for one (single, sole) distribution is no, the process is repeated for that data subset until there is a single distribution in the subset or a check 820 for whether a maximum depth is reached is yes. The maximum depth is how many times the distribution search is performed in a subsequence of the multi time series data, referencing to the concept of a tree’s maximum depth. The maximum depth limit is to avoid overfitting and promote continuous regimes that respect time dependency. Finally, the isolated splits from the multiple distributions are consolidated 822 to create the final training regimes, which capture, e.g., all the data distributions. The workflow 800 thereafter ends 823 in the example embodiment.
[0128] Consequently, the workflow 800 achieves an improved representation to train the anomaly detection model, that is, a ML model. The idea is to find all the data distribution that are present using an iterative process. During the process, the found distributions can be decomposed in smaller data distributions. The decomposition into smaller distributions is done one at the time. Therefore, after finding all the data distributions individually, the final process is to merge (join) them together and give the final recommended selected training data.
[0129] For the results of the present disclosure, assuming all the data comes from the same distribution, it is modeled with a GMM with a single component. If an assumption about the data distribution holds true, a simple train / test split of the sensor data would result in sets of data points in both training and testing that have the same Mahalanobis distance to the representative distribution. A Mahalanobis distance is a common measurement for distance, specifically, the distance from a point to another point, a given a distribution. The Mahalanobis distance may be used, according to an embodiment, to determine whether the selected training data is a single distribution that is sufficient for use as training data. Thus, if the Mahalanobis distance from one point to the location of a distribution increases, the probability of the point belonging to such a compared data distribution, decreases. A big distance means that the point has a small probability of belonging to that distribution. On the- 26 - 5083032. vlDocket No. 1086.2107001other hand, when the distances are significantly different, the methodology continues to the next step to search for different distributions on the data, as disclosed above with regard to the workflow 800 of FIG. 8.
[0130] FIG. 9 is a flow diagram of an example embodiment of a workflow 900 for finding data distributions in time series sensor data. In an embodiment, the method 900 is implemented at action 812 of the method 800. Continuing with FIG. 9, to find multiple distributions in the time series sensor data, the workflow 900 may begin 902, and proceed with action 904 that may, for non-limiting example, employ a GMM model on both training and test splits - increasing gradually a number of components of the GMM model and computing a silhouette score for each iteration (Kazor, K., Hering, A.S. (2015) Assessing the Performance of Model -Based Clustering Methods in Multivariate Time Series with Application to Identifying Regional Wind Regimes. JABES 20, 192-217, K. R. Shahapure and C. Nicholas, "Cluster Quality Analysis Using Silhouette Score, "2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), Sydney, NSW, Australia, 2020, 747-748).
[0131] An example embodiment may first test if the data belongs to a single distribution. If the data is from a single distribution, the methodology (workflow 900) may end (912). If, however, the data is not from one distribution, the methodology may proceed to find all the different distributions to create the splits and add data to the training set from all the different data distributions. Having all the different distributions in the data ensures that the training set represents all the data, which is a basic assumption for training ML models.
[0132] An increase in a number of components in the GMM model may be stopped when three consecutive scores are lower than the best score for non-limiting example, and the best number of distributions is the highest silhouette score. After determining the number of distributions in the data, the GMM model may be used to predict the distribution that each point belongs to. Nevertheless, industrial sensor data often contains noise, leading to a misclassification of certain points. To address this, at action 906 of the workflow 900, a static filtering process may be applied using a sliding window that calculates a statistical mode, effectively removing most of the misclassified points.
[0133] In some cases, however, residual misclassified points remain. To eliminate these, at action 908 of the workflow 900, a dynamic filter may be applied. This second filter, that is, the dynamic filter, may also use a sliding window to calculate the statistical mode, however, the dynamic filter incorporates a dynamic window size determined based on a longest- 27 - 5083032. vlDocket No. 1086.2107001duration of the misclassifications observed. Finally, at action 910 of the workflow 900, the number of points of each data distribution may be checked to ensure that each detected distribution contains enough points; such as a threshold total number of points, and regimes failing this criterion may be merged with a nearest valid distribution.
[0134] This multi-layer filtering approach ensures robust and accurate detection of valid regimes in noisy sensor data. Such filtering is part of the search to “find data distributions” disclosed by the actions of FIG. 9. Such process of “finding data distributions” is also referenced more generally in with regard to FIG. 8, above, which disclosed the general workflow 800. Continuing with reference to FIG. 9, the workflow 900 thereafter ends 912, following the action 910 in the example embodiment.Failure Signature Detection in the Automatic Selection of Data to Train ML Models
[0135] In some cases, it may be useful to forecast when a particular type of failure is going to occur in an industrial asset, also known as failure signature detection. In those cases, the automated selection of training data may include additional actions. For example, after finding the different distributions in the multivariate time-series data received from sensors, and adding such distributions to a training set as disclosed in the methodology above, an event with a target failure for detection may be added to the training set according to an example embodiment. In cases where there are multiple target failures, a simple rule of using oldest failure events for adding to the training set and using most recent failure events for adding to the test set may be employed.Automated selection of optimal method and set of hyperparameters
[0136] One important task associated with building a ML model is selecting a type of model. To create current APM anomaly detection models, following selection of training ranges, asset health monitoring system users may select a method type (e.g., ML model type) and a hyperparameters set. ML models often contain multiple hyperparameters, the values of which impact performance. Usually, data scientists must tune these hyperparameters to find models that work reasonably well. Yet, AMP solutions provide tools intended to be used by domain experts, rather than data scientists. Manual selection of method (ML model) type and- 28 - 5083032. vlDocket No. 1086.2107001hyperparameters is confusing for intended users of such software and leads to suboptimal performance of anomaly prediction models.
[0137] To simplify model creation, as well as to enhance quality of anomaly prediction, example embodiments disclosed herein may use hyperparameter optimization techniques instead of manual selection, thereby eliminating dual domain expertise in Engineering and Data Science that would otherwise be needed in order to use asset health monitoring systems effectively, and to further decrease, or eliminate, service team involvement in agent (ML model) building.
[0138] An example embodiment may employ BlendSearch (Wang, C., Wu, Q., Weimer, M., & Zhu, E. (2021). Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems, 3, 434-447) as a method to do hyperparameter optimization and model selection. Other methods can also be used, such as random search, Bayesian optimization, Particle Swarm Optimization, Genetic Algorithms, grid search, and dynamic search (Zong, B., Song, Q., Min, M. R., Cheng, W., Lumezanu, C., Cho, D., & Chen, H. (2018, February) Deep autoencoding gaussian mixture model for unsupervised anomaly detection.In International conference on learning representations), for non-limiting examples.
[0139] For non-limiting example, BlendSearch may be used because such method finds global minima fast and efficiently. BlendSearch achieves fast and global searching by combining local searching with global searching. BlendSearch leverages the frugality of CFO (Cost Frugal Optimization) (Jiang, X., Liu, K., Yan, J., & Chen, W. (2012). Application of improved SOM neural network in anomaly detection. Physics Procedia, 33, 1093-1099) and the space exploration capabilities of global search methods, such as Bayesian optimization. Like CFO, BlendSearch utilizes a low-cost initial point as input, if available, and starts the search from there. However, unlike CFO, BlendSearch does not wait for a local search to fully converge before trying new start points. These new start points are suggested by the global search method and filtered based on their distance to the existing points in the cost-related dimensions. BlendSearch gradually increases a trial cost and prioritizes among the global search thread and multiple local search threads based on optimism in the face of uncertainty.
[0140] An example embodiment of a workflow disclosed herein is also customizable. By "customizable," such workflow allows the addition of any custom model to the optimization process, as well as any custom metric. This flexibility allows an example embodiment of a workflow disclosed herein to be adapted to a wide range of use cases.- 29 - 5083032. vlDocketNo. 1086.2107001
[0141] To run a workflow disclosed herein, training ranges may be defined, a list of methods to select from may be provided, a list of hyperparameters to consider along with the search space of those hyperparameters may be employed, metrics to optimize on may be used, and a time budget a user is willing to wait may be employed as an evaluation criterion, for non-limiting examples.
[0142] An example embodiment may use training ranges that come from automated selection of training data, but users can override those recommendations with custom selection. A validation range may include everything not included in training data and not part of offline conditions (supplied by user).
[0143] An example embodiment of a methodology disclosed herein can search amongst a variety of ML models, including unsupervised and supervised models, and a variety of regression, classification, and clustering models. For each of the models a search space for the hyperparameters may be defined. Along with the search space, cost-related information may be provided whenever possible based on empirical knowledge and experiments.
[0144] To speed up computations, BlendSearch may use additional information about computation cost associated with certain hyperparameters / methods. Cost-related hyperparameters are a subset of the hyperparameters that directly affect the computation cost incurred in an evaluation of any hyperparameter configuration. Examples of those are a number of estimators and a maximum number of leaves for training cost of tree-based learners for non-limiting examples.
[0145] When cost-related hyperparameters exist, an evaluation cost in the search space is heterogeneous. BlendSearch search methods are designed to finish a tuning process at a low total cost when the evaluation cost in the search space is heterogeneous (Wang, C., Wu, Q., Weimer, M., & Zhu, E. (2021). Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems, 3, 434-447).
[0146] An example embodiment disclosed herein can also perform optimization using various metrics for classification, regression, and clustering workflows, such as: F-score, micro and macro F-scores, ROC AUC (Receiver Operating Characteristic - Area Under Curve), Log Loss (Logarithmic Loss), MAE (Mean Absolute Error), MSE (Mean Squared Error), R2(R-squared), MAPE (Mean Absolute Percentage Error), and / or RMSE (Root Mean Squared Error) for non-limiting examples. An example embodiment also supports the weighted versions of these metrics. An example embodiment disclosed herein allows the use of custom functions for any metric through user-defined functions (Miinz, G., Li, S., & Carle,- 30 - 5083032. vlDocket No. 1086.2107001G. (2007, September). Traffic anomaly detection using k-means clustering. In Gi / itg workshop mmbnet (Vol. 7, No. 9).
[0147] The default metric recommended for optimization of a workflow disclosed herein is the weighted Brier score (Rufibach, K. (2010). Use of Brier score to assess binary predictions. Journal of clinical epidemiology, 63(8), 938-939). This is one of the probabilistic metrics that are designed specifically to quantify the uncertainty in a classifier’s predictions. These are useful for problems for which there is less interest in incorrect vs. correct class predictions and more interest in the uncertainty the model has in predictions and penalizing those predictions that are wrong but highly confident. The Brier score measures the mean squared difference between the predicted probability and the actual outcome. Weights are assigned according to the prevalence of data points corresponding to labeled events vs. normal operations within the validation range. This is done to help with imbalance in the data, which is a widespread problem for anomaly detection tasks.
[0148] Alternatively, a linear combination of several metrics can be used for optimization, where a user supplies weights for the combination of the metrics. Automatically training an ML model may take longer than a traditional training procedure and, as such, an example embodiment may obtain input from a user on a time budget - the amount of time they are willing to wait - and may use such input as a constraint to the optimization process.
[0149] Based on such information input, disclosed above, an example embodiment of an optimization workflow may run using a selected metric(s). An example embodiment of a workflow for automated ML is disclosed below with regard to FIG.10, and generates a probability of a data point being an anomaly, while relying less on manual setup and fine tuning from a user.
[0150] FIG. 10 is a flow diagram of an example embodiment of a workflow (1000) for automatic (auto) ML. The workflow (1000) may begin (1002) and define cost for cost-related hyperparameters (1004) and define a search space for the hyperparameters (1010). The workflow (1000) may define training and validation ranges (1006), select metrics to optimize (1008), define constraint s) (e.g., time budget) (1012), and select an optimization method (1014). The workflow (1000) may optimize hyperparameters for a ML agent, e.g., an anomaly agent, and select a ML model, i.e., method, (1016) based on the training and validation ranges defined (1006), selected metrics (1008), defined search space (1010), defined constraints (1012), and / or selected optimization method (1014), and output an optimized anomaly agent (1017) (or other such ML model) that may, in turn, generate a- 31 - 5083032. vlDocketNo. 1086.2107001probability of a point being an anomaly (1018) (or other such ML model generated output). The workflow (1000) thereafter ends (1020) in the example embodiment.
[0151] FIG. 11 A is a plot 1100-A of an example embodiment of anomaly scores 1162 generated by a ML model tuned by a user. The anomaly scores 1162 are plotted over time 1131. The plot 1100-A shows that a confirmed anomaly (shaded area 1172) is not detected by an anomaly agent, created by the user, after spending 15 to 30 minutes.
[0152] FIG. 1 IB is a plot 1100-B of an example embodiment of anomaly scores 1162 generated by a ML model created by an Al-assisted workflow, such as the workflow (1000) of FIG. 10, disclosed above. Continuing with reference to FIG. 1 IB, the anomaly scores 1162 are plotted over time 1131. With reference to FIGS. 11 A and 1 IB, the plot 1100-B shows that the same event that caused the confirmed anomaly corresponding to the shaded area 1172 in the plot 1100-A of FIG. 11 A, is now correctly detected in the shaded region 1172 of plot 1100-B of FIG. 1 IB by an agent with optimized hyperparameters. In the example embodiment of FIG. 1 IB, it took under 2 minutes to run the hyperparameter optimization (HPO) that selected the best hyperparameters and model type.Detection of a need to retrain asset performance monitoring methods
[0153] FIG. 4, disclosed above, illustrates the pernicious effect of a failure to timely retrain ML models, which results in an alarm signal being in an incessant state of alarm. However, a question arises as to how a need to retrain a ML model could be detected, automatically. Practically, this could be accomplished in different ways, such as directly comparing the training data to prior states of operations. However, as dimensions increase, hypothesis tests that compare high-dimensional locations become increasingly less powerful. Therefore, it is better to compare a state of operations to what it was at a time of training in a lower dimensional representation. That could either be the lower dimensional statistic from which a probability signal is computed, or the probability signal itself.
[0154] A first component of the present disclosure is the application of statistical changepoint detection to a dimensionality reduced representation of sensor data. A wide variety of methods for changepoint detection have been proposed in the scientific literature, a non-exhaustive overview of which is presented in (Truong, C., Oudre, L., & Vayatis, N. (2020). Selective review of offline change point detection methods. Signal Processing, 167, 107299) for non-limiting example. However, not all changepoint detection methods are equally suitable to be deployed in a context of asset performance monitoring.- 32 - 5083032. vlDocket No. 1086.2107001
[0155] Changepoint detection methods perform what their name suggests: to identify a time point in a time series after which the time series behaves differently compared to a period before the change point. Different behavior is a general term that can imply changes in location (e.g., mean, which in one dimension looks like a ‘jump’), changes in variance (e.g., the signal is noisier than before), a combination of both, or an absence or presence of a trend in the time series for non-limiting examples. With this in mind, recall that an important purpose of asset performance monitoring tools described herein is to detect anomalies. Most changepoint detection methods would identify anomalies as changepoints, which would then produce a false detection of a need for an asset performance monitoring method (ML model) to be retrained. Therefore, it is useful that the changepoint method deployed in this context be insensitive to individual anomalies and only react to sustained changes in the underlying signal. In other words, the changepoint method needs to be statistically robust, which narrows down the number of valid approaches.
[0156] Different statistically robust changepoint detection estimators exist. At first, one can construct an M-estimator for changepoint detection by optimizing a robust loss function, such as a Huber or Hampel loss (Paul Fearnhead & Guillem Rigaill (2019) Changepoint Detection in the Presence of Outliers, Journal of the American Statistical Association, 114:525, 169-183) for non-limiting example. It is also possible to arrive at statistical robustness of the changepoint detection method by optimizing a non-parametric criterion based on signs or ranks (Lung-Yut-Fong, A., Levy-Leduc, C., & Cappe, O. (2015).Homogeneity and change-point detection tests for multivariate data using rank statistics. Journal de la societe franqaise de statistique, 156(A), 133-162), or by applying a smoothening kernel transformation to the data and performing the changepoint detection in the transformed space (Celisse, A., Marot, G., Pierre-Jean, M., & Rigaill, G. J. (2018). New efficient algorithms for multiple change-point detection with reproducing kernels.Computational Statistics & Data Analysis, 128, 200-220, Arlot, S., Celisse, A., & Harchaoui, Z. (2019). A kernel multiple change-point algorithm via model selection. Journal of Machine Learning Research, 20 (162), 1-56), for non-limiting examples. Given a suitable selection of hyperparameters, each of these approaches could be adopted to accomplish detection of sustained changes in the signal of an asset performance monitoring agent, such as an anomaly detection agent or a failure detection agent implemented by a ML model that may detect a changepoint robustly, such as the changepoint disclosed below with regard to FIG. 12 for non-limiting example.- 33 - 5083032. vlDocketNo. 1086.2107001
[0157] FIG. 12 is a plot 1200 of a univariate signal 1262 which is identical to the univariate signal 433 plotted in FIG. 4, disclosed above. With reference to FIG. 4 and FIG. 12, the plot 1200 is for the same time series disclosed above with reference to FIG. 4. In the plot 1200, a background changes in a region 1272 that follows a detected changepoint 1274.
[0158] It should be noted that changepoint detection methods will detect all jumps in the objective considered. In the example embodiment shown in FIG. 4, disclosed above, the objective was to detect a change in location of the time series (which is the most common objective in changepoint detection). The changepoint detection method correctly identified a jump. However, in many cases, jumps may not be alarming in each direction. For instance, an anomaly detection agent that falsely alerts the operators in normal modes of operation, as the one plotted in FIG. 4 does, requires action.
[0159] However, if the signal in FIG. 4 and FIG. 12 had jumped to a lower baseline, no alerts would be produced and, in fact, such a jump would not warrant action. Therefore, it is useful for changepoint detection in the area of asset performance management to advantageously only detect changepoints in signals that correspond to moves in one direction. Such detection may be accomplished by adapting an optimization criterion in changepoint detection to a unidirectional one, or simply by performing bidirectional changepoint detection, followed by a statistical sign test for non-limiting example.
[0160] Once a need to retrain an asset performance monitoring agent, such as a ML model, has been detected, as is the case for the univariate signal 433 plotted in FIG. 4 and the univariate signal 1262 plotted against time 1231 in FIG. 12, the process returns as disclosed with regard to the workflow 200 of FIG. 2, disclosed above, and starts over.
[0161] As disclosed above, novel example embodiments of a workflow disclosed herein allow deployment of ML models for asset performance management that can maintain a constant state of optimality to represent an actual mode of operations. As opposed to older workflows, no intervention is needed by an expert to select training data, to fine-tune model hyperparameters, or to identify that a ML model is no longer valid and that selection of training data and fine-tuning of the model parameters would have to be repeated. Example embodiments of computer-implemented methods for implementing such workflow are disclosed below.
[0162] FIG. 13 is a flow diagram of an example embodiment of a computer-implemented method (1300) for automating ML model retraining detection. The computer-implemented method (1300) begins (1302) and may comprise receiving sensor data from at least one- 34 - 5083032. vlDocket No. 1086.2107001sensor coupled to a real-world system (1304). The sensor data may be employed by a ML model trained to monitor performance of the real-world system. A current output of the ML model, determined by the ML model based on processing the sensor data received, may indicate current performance of the real-world system. The computer-implemented method (1300) may further comprise identifying historical training data used to train the ML model and a historical output of the ML model, indicating historical performance of the real-world system, determined by the ML model based on processing the historical training data identified (1306). The computer-implemented method (1300) may further comprise determining whether there is at least one sustained change between the sensor data received and the historical training data identified based on a comparison between the current output of the ML model and the historical output of the ML model (1308). The computer-implemented method (1300) may further comprise, responsive to determining that there is at least one sustained change, automatically detecting that the ML model should be retrained (1310). The computer-implemented method (1300) thereafter ends (1312) in the example embodiment.
[0163] The computer-implemented method (1300) may further comprise retraining the ML model responsive to automatically detecting that the ML model should be retrained. The retraining may employ at least a portion of the sensor data received.
[0164] In an embodiment of the computer-implemented method (1300), a) the sensor data received may represent time-series sensor data, b) the historical training data identified may represent time-series historical training data, or c) any combination of a) and b).
[0165] In an example embodiment of the method 1300, a) the time series sensor data may be non-stationary multivariate time series sensor data, b) the time series historical training data may be non-stationary multivariate time series historical training data, or any combination of a) and b).
[0166] The identifying (1306) may include a) searching a look-up table (LUT) in a database, b) selecting the historical training data based on a least one training range, or c) any combination of a) and b).
[0167] The determining (1308) may include employing a statistical changepoint detection method. The statistical changepoint detection method may be insensitive to individual anomalies between the current output of the ML model and the historical output of the ML model.- 35 - 5083032. vlDocketNo. 1086.2107001
[0168] The sensor data received (1304) may be associated with a plant process, an electrical power grid operation process, a hydrocarbon exploration process, a hydrocarbon production process, a mining operation process, a process of the real-world system, or any processes or combination of processes of the real-world system for non-limiting examples.
[0169] The sensor data received (1304) may be industrial sensor data for non-limiting example. The real-world system may be an industrial, real-world system for non-limiting example.
[0170] FIG. 14 is a flow diagram of an example embodiment of a computer-implemented method (1400) for automating ML model selection. The computer-implemented method (1400) begins (1402) and may comprise receiving at least one criterion (1404). The computer-implemented method (1400) may further comprise iteratively evaluating each candidate ML model of a plurality of candidate ML models until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model of the given iteration meets the at least one criterion and ii) evaluating the candidate ML model of an iteration subsequent to the given iteration exceeds an evaluation limit (1406). Candidate ML models for evaluating may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be a selected optimum ML model. The computer-implemented method (1400) thereafter ends (1408) in the example embodiment.
[0171] The at least one criterion received (1404) may include a) at least one training range, b) a list of hyperparameters for a hyperparameter search space, c) at least one optimization metric, d) a time budget for the iteratively evaluating, or e) any combination of a)-d).
[0172] The computer-implemented method (1400) may further comprise monitoring performance of at least one asset in a real-world system using the selected optimum ML model.
[0173] The iteratively evaluating (1406) may include employing a) a BlendSearch method, b) a random search method, c) a Bayesian optimization method, d) a particle swarm optimization method, e) a genetic method, f) a grid search method, g) a dynamic search method, h) other search method, or i) any combination of a)-h).
[0174] The plurality of candidate ML models may include a) an unsupervised ML model, b) a supervised ML model, c) a regression ML model, d) a classification ML model, e) a clustering ML model, or f) any combination of a)-e).- 36 - 5083032. vlDocketNo. 1086.2107001
[0175] The iteratively evaluating (1406) may include employing at least one metric. A metric of the at least one metric may include a classification metric, a regression metric, or a clustering metric for non-limiting examples. The iteratively evaluating (1406) may include employing a plurality of metrics with respective weights.
[0176] FIG. 15 is a flow diagram of an example embodiment of a computer-implemented method (1500) for automating ML model hyperparameter selection. The computer-implemented method (1500) begins (1502) and may comprise receiving at least one criterion and a hyperparameter search space (1504). The computer-implemented method (1500) may further comprise iteratively evaluating hyperparameter values, within the hyperparameter search space received, for hyperparameters of a ML model until i) a result of the iteratively evaluating of a given iteration indicates that hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit (1506). Hyperparameter values for evaluating may be selected based on learned performance of the ML model and learned computational cost of the evaluating. The hyperparameter values of the given iteration may be selected optimum hyperparameters for the ML model. The computer-implemented method (1500) thereafter ends (1508) in the example embodiment.
[0177] FIG. 16 is a flow diagram of an example embodiment of a computer-implemented method (1600) for automating ML model and ML model hyperparameter selection. The computer-implemented method (1600) begins (1602) and may comprise receiving at least one criterion and a hyperparameter search space (1604). The computer-implemented method (1600) may further comprise iteratively evaluating each candidate ML model of a plurality of candidate ML models and hyperparameter values, within the hyperparameter search space received, for hyperparameters of each candidate ML model until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model and hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating the candidate ML model and hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit (1606). Candidate ML models and hyperparameter values for evaluating may be selected based on learned performance of the candidate ML models and learned computational cost of the evaluating. The candidate ML model of the given iteration may be a selected optimum ML model and the hyperparameter values of the given iteration may be selected optimum hyperparameter values. The computer-implemented method (1600) thereafter ends (1608) in the example embodiment.- 37 - 5083032. vlDocket No. 1086.2107001
[0178] FIG. 17 is a flow diagram of an example embodiment of a computer-implemented method (1700) for automating ML model training data selection. The computer-implemented method (1700) begins (1702) and may comprise receiving a time-series dataset for monitoring performance of an asset of a real-world system (1704). The computer-implemented method (1700) may further comprise producing a reduced time-series dataset by excluding a portion of data of the time-series dataset received (1706). The computer-implemented method (1700) may further comprise iteratively processing the reduced timeseries dataset to identify at least one data distribution in the reduced time-series dataset (1708). The computer-implemented method (1700) may further comprise automatically selecting a given data distribution from the at least one data distribution identified, wherein the given data distribution selected is selected as training data for a ML model and the given data distribution is selected automatically based on a temporal correspondence between the given data distribution selected and a target timeframe for monitoring performance of the asset with the ML model (1710). The computer-implemented method (1700) thereafter ends (1712) in the example embodiment.
[0179] Selecting (1710) the given data distribution may comprise detecting anomalies or detecting failure signatures. The given data distribution may include at least one data distribution.
[0180] Selecting (1710) the given data distribution may include identifying a target behavior in the given data distribution and automatically selecting the given data distribution based on a) the temporal correspondence between the given data distribution selected and the target timeframe and b) the identifying the target behavior in the given data distribution.
[0181] The target behavior may include a) at least one anomaly, b) at least one failure signature, or c) any combination of a) and b). Further, as disclosed above, In some cases, it may be useful to forecast when a particular type of failure is going to occur in an industrial asset, also known as failure signature detection. In those cases, the ML model training data selection may include additional actions. For example, after finding the different distributions in the multivariate time series data received from the sensors, and adding such distributions to the training set as disclosed in the methodology above, the event with the target failure to detect may be added to a training set according to an example embodiment. In cases where there are multiple target failures, a simple rule of using oldest failure events for adding to the training set and using most recent failure events for adding to a test set may be employed.- 38 - 5083032. vlDocketNo. 1086.2107001
[0182] Continuing with reference to FIG. 17, the computer-implemented method (1700) may further comprise automatically training the ML model using the given data distribution selected (1710) automatically. The training may produce a trained ML model. The computer-implemented method (1700) may further comprise deploying the trained ML model in the target timeframe.
[0183] The portion of data excluded (1706) may correspond to a) a time period for which the asset was known to be offline, b) a time period for which the asset was associated with a confirmed failure event, or c) any combination of a) and b).
[0184] The time-series dataset received may, for non-limiting example, be sensor data received from at least one sensor coupled to the asset. The time-series dataset may include non- stationary multivariate time series data or stationary multivariate time series data for non-limiting examples.
[0185] The given data distribution may be selected (1710) based on the temporal correspondence and a determination that the given data distribution, in its entirety, encompasses a training data portion and a test data portion in a single distribution.
[0186] The iteratively processing (1708) may include determining whether the reduced time-series dataset, in its entirety, corresponds to one distribution. Responsive to determining that the reduced time-series dataset, in its entirety, does not correspond to one distribution, the iteratively processing (1708) may further include: iteratively i) dividing the reduced timeseries dataset into a plurality of datasets, the iteratively dividing including dividing each dataset of the plurality of datasets into a candidate training data portion and test data portion and ii) evaluating whether each dataset of the plurality of datasets corresponds to one distribution, until a) each dataset of the plurality of datasets corresponds to a respective one distribution or b) a total number of maximum iterations has been performed. The at least one data distribution identified may be composed of each candidate training data portion.
[0187] The computer-implemented methods of the present disclosure (e.g., computer-implemented methods 1300, 1400, 1500, 1600, 1700) may be implemented via a computer with an internal structure, such as disclosed below for non-limiting example.
[0188] FIG. 18 is a block diagram of an example of an internal structure of a computer 1800 in which various embodiments of the present disclosure may be implemented. The computer 1800 contains a system bus 1801, where a bus is a set of hardware lines used for data transfer among the components of a computer or digital processing system. The system bus 1801 is essentially a shared conduit that connects different elements of a computer- 39 - 5083032. vlDocket No. 1086.2107001system (e.g., processor, disk storage, memory, input / output ports, network ports, etc.) that enables the transfer of information between the elements. Coupled to the system bus 1801 is an I / O device interface 1803 for connecting various input and output devices (e.g., keyboard, mouse, display monitors, printers, speakers, microphone, etc.) to the computer 1800. A network interface 1805 allows the computer 1800 to connect to various other devices attached to a network (e.g., global computer network, wide area network, local area network, etc.). Memory 1807 provides volatile or non-volatile storage for computer software instructions 1809 and data 1811 that may be used to implement embodiments (e.g., computer-implemented methods 1300, 1400, 1500, 1600, 1700, and workflows 200, 800, 900, 1000) of the present disclosure, where the volatile and non-volatile memories are examples of non-transitory media. Disk storage 1813 also provides non-volatile storage for the computer software instructions 1809 and data 1811 that may be used to implement embodiments (e.g., computer-implemented methods 1300, 1400, 1500, 1600, 1700, and workflows 200, 800, 900, 1000) of the present disclosure. A central processor unit 1817 is also coupled to the system bus 1801 and provides for the execution of computer instructions.
[0189] Further example embodiments disclosed herein may be configured using a computer program product; for example, controls may be programmed in software for implementing example embodiments. Further example embodiments may include a non-transitory computer-readable-medium that contains instructions that may be executed by a processor, and, when loaded and executed, cause the processor to complete methods and techniques described herein. It should be understood that elements of the block and flow diagrams may be implemented in software or hardware, such as via one or more arrangements of circuitry of FIG. 18, disclosed above, or equivalents thereof, firmware, a combination thereof, or other similar implementation determined in the future.
[0190] In addition, the elements of the block and flow diagrams described herein may be combined or divided in any manner in software, hardware, or firmware. If implemented in software, the software may be written in any language that can support the example embodiments disclosed herein. The software may be stored in any form of computer readable medium, such as random-access memory (RAM), read only memory (ROM), compact disk read-only memory (CD-ROM), and so forth. In operation, a general purpose or applicationspecific processor or processing core loads and executes software in a manner well understood in the art. It should be understood further that the block and flow diagrams may include more or fewer elements, be arranged or oriented differently, or be represented- 40 - 5083032. vlDocket No. 1086.2107001differently. It should be understood that implementation may dictate the block, flow, and / or network diagrams and the number of block and flow diagrams illustrating the execution of embodiments disclosed herein.
[0191] The teachings of all patents, published applications, and references cited herein are incorporated by reference in their entirety.
[0192] While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments encompassed by the appended claims. For instance, while embodiments are described herein as being implemented in relation to ML models directed toward particular tasks, e.g., anomaly detection, embodiments are not so limited and, instead, embodiments can be applied to any ML models being utilized for any tasks.
[0193] As noted above, embodiments may include deploying trained ML models.Deploying a trained ML model may include the ML model receiving data, e.g., from a sensor coupled to an asset, e.g., a wind turbine, and, in turn, the ML model processing the received data to generate an analysis result, e.g., an indication that the turbine is entering an anomalous state of operation. This analysis result can, in turn, cause automated actions to be taken in the real -world. For example, operating settings for the asset may be modified to prevent the asset from entering the anomalous state or the asset may be taken offline to avoid damage to the asset during the period of anomalous operation.5083032. vl
Claims
Docket No. 1086.2107001CLAIMSWhat is claimed is:
1. A computer-implemented method for automating machine-learning (ML) model retraining detection, the computer-implemented method comprising:receiving sensor data from at least one sensor coupled to a real-world system, the sensor data employed by a ML model trained to monitor performance of the real- world system, a current output of the ML model, determined by the ML model based on processing the sensor data received, indicating current performance of the real- world system;identifying historical training data used to train the ML model and a historical output of the ML model, indicating historical performance of the real-world system, determined by the ML model based on processing the historical training data identified;determining whether there is at least one sustained change between the sensor data received and the historical training data identified based on a comparison between the current output of the ML model and the historical output of the ML model; andresponsive to determining that there is at least one sustained change, automatically detecting that the ML model should be retrained.
2. The computer-implemented method of Claim 1, further comprising retraining the ML model responsive to automatically detecting that the ML model should be retrained, the retraining employing at least a portion of the sensor data received.
3. The computer-implemented method of Claim 1 or 2, wherein a) the sensor data received represents time-series sensor data, b) the historical training data identified represents time-series historical training data, or c) any combination of a) and b).
4. The computer-implemented method of any of Claims 1-3, wherein the identifying includes a) searching a look-up table (LUT) in a database, b) selecting the historical training data based on a least one training range, or c) any combination of a) and b).
5. The computer-implemented method of any of Claims 1-4, wherein the determining includes employing a statistical changepoint detection method and wherein the- 42 - 5083032. vlDocket No. 1086.2107001statistical changepoint detection method is insensitive to individual anomalies between the current output of the ML model and the historical output of the ML model.
6. The computer-implemented method of any of Claims 1-5, wherein the sensor data received is associated with a plant process, an electrical power grid operation process, a hydrocarbon exploration process, a hydrocarbon production process, a mining operation process, a process of the real-world system, or any combination of processes of the real-world system.
7. The computer-implemented method of any of Claims 1-6, wherein the sensor data received is industrial sensor data and wherein the real-world system is an industrial, real -world system.
8. A computer-implemented method for automating machine-learning (ML) model selection, the computer-implemented method comprising:receiving at least one criterion; anditeratively evaluating each candidate ML model of a plurality of candidate ML models until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model of the given iteration meets the at least one criterion and ii) evaluating the candidate ML model of an iteration subsequent to the given iteration exceeds an evaluation limit, whereincandidate ML models for evaluating are selected based on learned performance of the candidate ML models and learned computational cost of the evaluating, andthe candidate ML model of the given iteration is a selected optimum ML model.
9. The computer-implemented method of Claim 8, wherein the at least one criterion received includes a) at least one training range, b) a list of hyperparameters for a hyperparameter search space, c) at least one optimization metric, d) a time budget for the iteratively evaluating, or e) any combination of a)-d).- 43 - 5083032. vlDocket No. 1086.210700110. The computer-implemented method of Claim 8 or 9, further comprising monitoring performance of at least one asset in a real-world system using the selected optimum ML model.
11. The computer-implemented method of any of Claims 8-10, wherein the iteratively evaluating includes employing a) a BlendSearch method, b) a random search method, c) a Bayesian optimization method, d) a particle swarm optimization method, e) a genetic method, f) a grid search method, g) a dynamic search method, h) other search method, or i) any combination of a)-h).
12. The computer-implemented method of any of Claims 8-11, wherein the plurality of candidate ML models includes a) an unsupervised ML model, b) a supervised ML model, c) a regression ML model, d) a classification ML model, e) a clustering ML model, or f) any combination of a)-e).
13. The computer-implemented method of any of Claims 8-12, wherein the iteratively evaluating includes employing at least one metric and wherein a metric of the at least one metric includes a classification metric, a regression metric, or a clustering metric.
14. The computer-implemented method of any of Claims 8-13, wherein the iteratively evaluating includes employing a plurality of metrics with respective weights.
15. A computer-implemented method for automating machine-learning (ML) model hyperparameter selection, the computer-implemented method comprising:receiving at least one criterion and a hyperparameter search space; iteratively evaluating hyperparameter values, within the hyperparameter search space received, for hyperparameters of a ML model until i) a result of the iteratively evaluating of a given iteration indicates that hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit, wherein hyperparameter values for evaluating are selected based on learned performance of the ML model and learned computational cost of the evaluating; andthe hyperparameter values of the given iteration are selected optimum hyperparameters for the ML model.- 44 - 5083032. vlDocket No. 1086.210700116. A computer-implemented method for automating machine-learning (ML) model and ML model hyperparameter selection, the computer-implemented method comprising:receiving at least one criterion and a hyperparameter search space; and iteratively evaluating each candidate ML model of a plurality of candidate ML models and hyperparameter values, within the hyperparameter search space received, for hyperparameters of each candidate ML model until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model and hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating the candidate ML model and hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit, whereincandidate ML models and hyperparameter values for evaluating are selected based on learned performance of the candidate ML models and learned computational cost of the evaluating, andthe candidate ML model of the given iteration is a selected optimum ML model and the hyperparameter values of the given iteration are selected optimum hyperparameter values.
17. A computer-implemented method for automating machine-learning (ML) model training data selection, the computer-implemented method comprising:receiving a time-series dataset for monitoring performance of an asset of a real -world system;producing a reduced time-series dataset by excluding a portion of data of the time-series dataset received;iteratively processing the reduced time-series dataset to identify at least one data distribution in the reduced time-series dataset; andautomatically selecting a given data distribution from the at least one data distribution identified, wherein the given data distribution selected is selected as training data for a ML model and the given data distribution is selected automatically based on a temporal correspondence between the given data distribution selected and a target timeframe for monitoring performance of the asset with the ML model.
18. The computer-implemented method of Claim 17, wherein selecting the given data distribution includes:identifying a target behavior in the given data distribution; and- 45 - 5083032. vlDocket No. 1086.2107001automatically selecting the given data distribution based on a) the temporal correspondence between the given data distribution selected and the target timeframe and b) the identifying the target behavior in the given data distribution.
19. The computer-implemented of Claim 18, wherein the target behavior includes a) at least one anomaly, b) at least one failure signature, or c) any combination of a) and b).
20. The computer-implemented method of any of Claims 17-19, further comprising:training the ML model using the given data distribution selected automatically, the training producing a trained ML model; anddeploying the trained ML model in the target timeframe.
21. The computer-implemented method of any of Claims 17-20, wherein the portion of data excluded corresponds to a) a time period for which the asset was known to be offline, b) a time period for which the asset was associated with a confirmed failure event, or c) any combination of a) and b).
22. The computer-implemented method of any of Claims 17-21, wherein the time-series dataset received is sensor data received from at least one sensor coupled to the asset.
23. The computer-implemented method of any of Claims 17-22, wherein the time-series dataset includes non-stationary multivariate time series data or stationary multivariate time series data.
24. The computer-implemented method of any of Claims 17-23, wherein the given data distribution is selected based on the temporal correspondence and a determination that the given data distribution, in its entirety, encompasses a training data portion and a test data portion in a single distribution.
25. The computer-implemented method of any of Claims 17-24, wherein the iteratively processing includes:determining whether the reduced time-series dataset, in its entirety, corresponds to one distribution and, responsive to determining that the reduced timeseries dataset, in its entirety, does not correspond to one distribution, the iteratively processing further includes:- 46 - 5083032. vlDocket No. 1086.2107001iteratively i) dividing the reduced time-series dataset into a plurality of datasets, the iteratively dividing including dividing each dataset of the plurality of datasets into a candidate training data portion and test data portion and ii) evaluating whether each dataset of the plurality of datasets corresponds to one distribution, until a) each dataset of the plurality of datasets corresponds to a respective one distribution or b) a total number of maximum iterations has been performed, wherein the at least one data distribution identified is composed of each candidate training data portion.
26. A computer-based system for automating machine-learning (ML) model retraining detection, the computer-based system comprising:at least one processor; andat least one memory with computer code instructions stored thereon, the at least one processor and the at least one memory, with the computer code instructions, being configured to cause the computer-based system to:receive sensor data from at least one sensor coupled to a real-world system, the sensor data employed by a ML model trained to monitor performance of the real-world system, a current output of the ML model, determined by the ML model based on processing the sensor data received, indicating current performance of the real-world system;identify historical training data used to train the ML model and a historical output of the ML model, indicating historical performance of the real-world system, determined by the ML model based on processing the historical training data identified;determine whether there is at least one sustained change between the sensor data received and the historical training data identified based on a comparison between the current output of the ML model and the historical output of the ML model; andresponsive to determining that there is at least one sustained change, automatically detect that the ML model should be retrained.
27. The computer-based system of Claim 26, wherein the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer-based system to retrain the ML model responsive to automatically- 47 - 5083032. vlDocket No. 1086.2107001detecting that the ML model should be retrained, the retraining employing at least a portion of the sensor data received.
28. The computer-based system of Claim 26 or 27, wherein a) the sensor data received represents time-series sensor data, b) the historical training data identified represents time-series historical training data, or c) any combination of a) and b).
29. The computer-based system of any of Claims 26-28, wherein, to identify the historical training data used to train the ML model and the historical output of the ML model, the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer-based system to a) search a look-up table (LUT) in a database, b) select the historical training data based on a least one training range, or c) any combination of a) and b).
30. The computer-based system of any of Claims 26-29, wherein, to determine whether there is at least one sustained change between the sensor data received and the historical training data identified, the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer-based system to employ a statistical changepoint detection method and wherein the statistical changepoint detection method is insensitive to individual anomalies between the current output of the ML model and the historical output of the ML model.
31. The computer-based system of any of Claims 26-30, wherein the sensor data received is associated with a plant process, an electrical power grid operation process, a hydrocarbon exploration process, a hydrocarbon production process, a mining operation process, a process of the real-world system, or any combination of processes of the real-world system.
32. The computer-based system of any of Claims 26-31, wherein the sensor data received is industrial sensor data and wherein the real-world system is an industrial, real-world system.
33. A computer-based system for automating machine-learning (ML) model selection, the computer-based system comprising:- 48 - 5083032. vlDocket No. 1086.2107001at least one processor; andat least one memory with computer code instructions stored thereon, the at least one processor and the at least one memory, with the computer code instructions, being configured to cause the computer-based system to:receive at least one criterion; anditeratively evaluate each candidate ML model of a plurality of candidate ML models until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model of the given iteration meets the at least one criterion and ii) evaluating the candidate ML model of an iteration subsequent to the given iteration exceeds an evaluation limit, wherein candidate ML models for evaluating are selected based on learned performance of the candidate ML models and learned computational cost of the evaluating, andthe candidate ML model of the given iteration is a selected optimum ML model.
34. The computer-based system of Claim 33, wherein the at least one criterion received includes a) at least one training range, b) a list of hyperparameters for a hyperparameter search space, c) at least one optimization metric, d) a time budget for the iteratively evaluating, or e) any combination of a)-d).
35. The computer-based system of Claim 33 or 34, wherein the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer-based system to monitor performance of at least one asset in a real -world system using the selected optimum ML model.
36. The computer-based system of any of Claims 33-35, wherein, to iteratively evaluate each candidate ML model of a plurality of candidate ML models, the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer-based system to employ a) a BlendSearch method, b) a random search method, c) a Bayesian optimization method, d) a particle swarm optimization method, e) a genetic method, f) a grid search method, g) a dynamic search method, h) other search method, or i) any combination of a)-h).- 49 - 5083032. vlDocket No. 1086.210700137. The computer-based system of any of Claims 33-36, wherein the plurality of candidate ML models includes a) an unsupervised ML model, b) a supervised ML model, c) a regression ML model, d) a classification ML model, e) a clustering ML model, or f) any combination of a)-e).
38. The computer-based system of any of Claims 33-37, wherein, to iteratively evaluate each candidate ML model of a plurality of candidate ML models, the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer-based system to employ at least one metric and wherein a metric of the at least one metric includes a classification metric, a regression metric, or a clustering metric.
39. The computer-based system of any of Claims 33-38, wherein, to iteratively evaluate each candidate ML model of a plurality of candidate ML models, the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer-based system to employ a plurality of metrics with respective weights.
40. A computer-based system for automating machine-learning (ML) model hyperparameter selection, the computer-based system comprising:at least one processor; andat least one memory with computer code instructions stored thereon, the at least one processor and the at least one memory, with the computer code instructions, being configured to cause the computer-based system to:receive at least one criterion and a hyperparameter search space; iteratively evaluate hyperparameter values, within the hyperparameter search space received, for hyperparameters of a ML model until i) a result of the iteratively evaluating of a given iteration indicates that hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit, whereinhyperparameter values for evaluating are selected based on learned performance of the ML model and learned computational cost of the evaluating; and- 50 - 5083032. vlDocket No. 1086.2107001the hyperparameter values of the given iteration are selected optimum hyperparameters for the ML model.
41. A computer-based system for automating machine-learning (ML) model and ML model hyperparameter selection, the computer-based system comprising:at least one processor; andat least one memory with computer code instructions stored thereon, the at least one processor and the at least one memory, with the computer code instructions, being configured to cause the computer-based system to:receive at least one criterion and a hyperparameter search space; and iteratively evaluate each candidate ML model of a plurality of candidate ML models and hyperparameter values, within the hyperparameter search space received, for hyperparameters of each candidate ML model until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model and hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating the candidate ML model and hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit, whereincandidate ML models and hyperparameter values for evaluating are selected based on learned performance of the candidate ML models and learned computational cost of the evaluating, andthe candidate ML model of the given iteration is a selected optimum ML model and the hyperparameter values of the given iteration are selected optimum hyperparameter values.
42. A computer-based system for automating machine-learning (ML) model training data selection, the computer-based system comprising:at least one processor; andat least one memory with computer code instructions stored thereon, the at least one processor and the at least one memory, with the computer code instructions, being configured to cause the computer-based system to:receive a time-series dataset for monitoring performance of an asset of a real -world system;- 51 - 5083032. vlDocket No. 1086.2107001produce a reduced time-series dataset by excluding a portion of data of the time-series dataset received;iteratively process the reduced time-series dataset to identify at least one data distribution in the reduced time-series dataset; andautomatically select a given data distribution from the at least one data distribution identified, wherein the given data distribution selected is selected as training data for a ML model and the given data distribution is selected automatically based on a temporal correspondence between the given data distribution selected and a target timeframe for monitoring performance of the asset with the ML model.
43. The computer-based system of Claim 42, wherein, to select the given data distribution, the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer-based system to:identify a target behavior in the given data distribution; andautomatically select the given data distribution based on a) the temporal correspondence between the given data distribution selected and the target timeframe and b) the identifying the target behavior in the given data distribution.
44. The computer-based system of Claim 43, wherein the target behavior includes a) at least one anomaly, b) at least one failure signature, or c) any combination of a) and b).
45. The computer-based system of any of Claims 42-44, wherein the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer-based system to:train the ML model using the given data distribution selected automatically, the training producing a trained ML model; anddeploy the trained ML model in the target timeframe.
46. The computer-based system of any of Claims 42-45, wherein the portion of data excluded corresponds to a) a time period for which the asset was known to be offline, b) a time period for which the asset was associated with a confirmed failure event, or c) any combination of a) and b).- 52 - 5083032. vlDocket No. 1086.210700147. The computer-based system of any of Claims 42-46, wherein the time-series dataset received is sensor data received from at least one sensor coupled to the asset.
48. The computer-based system of any of Claims 42-47, wherein the time-series dataset includes non- stationary multivariate time series data or stationary multivariate time series data.
49. The computer-based system of any of Claims 42-48, wherein the given data distribution is selected based on the temporal correspondence and a determination that the given data distribution, in its entirety, encompasses a training data portion and a test data portion in a single distribution.
50. The computer-based system of any of Claims 42-49, wherein, to iteratively process the reduced time-series dataset to identify the at least one data distribution in the reduced time-series dataset, the at least one processor and the at least one memory, with the computer code instructions, are further configured to cause the computer- based system to:determine whether the reduced time-series dataset, in its entirety, corresponds to one distribution and, responsive to determining that the reduced time-series dataset, in its entirety, does not correspond to one distribution; anditeratively i) divide the reduced time-series dataset into a plurality of datasets, the iteratively dividing including dividing each dataset of the plurality of datasets into a candidate training data portion and test data portion and ii) evaluate whether each dataset of the plurality of datasets corresponds to one distribution, until a) each dataset of the plurality of datasets corresponds to a respective one distribution or b) a total number of maximum iterations has been performed, wherein the at least one data distribution identified is composed of each candidate training data portion.
51. A non-transitory computer-readable medium for automating machine-learning (ML) model retraining detection, the non-transitory computer-readable medium having encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to:receive sensor data from at least one sensor coupled to a real-world system, the sensor data employed by a ML model trained to monitor performance of the real-- 53 - 5083032. vlDocket No. 1086.2107001world system, a current output of the ML model, determined by the ML model based on processing the sensor data received, indicating current performance of the real- world system;identify historical training data used to train the ML model and a historical output of the ML model, indicating historical performance of the real-world system, determined by the ML model based on processing the historical training data identified;determine whether there is at least one sustained change between the sensor data received and the historical training data identified based on a comparison between the current output of the ML model and the historical output of the ML model; andresponsive to determining that there is at least one sustained change, automatically detect that the ML model should be retrained.
52. The non-transitory computer-readable medium of Claim 51, wherein the sequence of instructions, when loaded and executed by the at least one processor, further causes the at least one processor to retrain the ML model responsive to automatically detecting that the ML model should be retrained, the retraining employing at least a portion of the sensor data received.
53. The non-transitory computer-readable medium of Claim 51 or 52, wherein a) the sensor data received represents time-series sensor data, b) the historical training data identified represents time-series historical training data, or c) any combination of a) and b).
54. The non-transitory computer-readable medium of any of Claims 51-53, wherein, to identify the historical training data used to train the ML model and the historical output of the ML model, the sequence of instructions, when loaded and executed by the at least one processor, causes the at least one processor to a) search a look-up table (LUT) in a database, b) select the historical training data based on a least one training range, or c) any combination of a) and b).
55. The non-transitory computer-readable medium of any of Claims 51-54, wherein, to determine whether there is at least one sustained change between the sensor data- 54 - 5083032. vlDocket No. 1086.2107001received and the historical training data identified, the sequence of instructions, when loaded and executed by the at least one processor, causes the at least one processor to employ a statistical changepoint detection method and wherein the statistical changepoint detection method is insensitive to individual anomalies between the current output of the ML model and the historical output of the ML model.
56. The non-transitory computer-readable medium of any of Claims 51-55, wherein the sensor data received is associated with a plant process, an electrical power grid operation process, a hydrocarbon exploration process, a hydrocarbon production process, a mining operation process, a process of the real-world system, or any combination of processes of the real-world system.
57. The non-transitory computer-readable medium of any of Claims 51-56, wherein the sensor data received is industrial sensor data and wherein the real-world system is an industrial, real-world system.
58. A non-transitory computer-readable medium for automating machine-learning (ML) model selection, the non-transitory computer-readable medium having encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to:receive at least one criterion; anditeratively evaluate each candidate ML model of a plurality of candidate ML models until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model of the given iteration meets the at least one criterion and ii) evaluating the candidate ML model of an iteration subsequent to the given iteration exceeds an evaluation limit, whereincandidate ML models for evaluating are selected based on learned performance of the candidate ML models and learned computational cost of the evaluating, andthe candidate ML model of the given iteration is a selected optimum ML model.
59. The non-transitory computer-readable medium of Claim 58, wherein the at least one criterion received includes a) at least one training range, b) a list of hyperparameters- 55 - 5083032. vlDocket No. 1086.2107001for a hyperparameter search space, c) at least one optimization metric, d) a time budget for the iteratively evaluating, or e) any combination of a)-d).
60. The non-transitory computer-readable medium of Claim 58 or 59, wherein the sequence of instructions, when loaded and executed by the at least one processor, further causes the at least one processor to monitor performance of at least one asset in a real -world system using the selected optimum ML model.
61. The non-transitory computer-readable medium of any of Claims 58-60, wherein, to iteratively evaluate each candidate ML model of a plurality of candidate ML models, the sequence of instructions, when loaded and executed by the at least one processor, causes the at least one processor to employ a) a BlendSearch method, b) a random search method, c) a Bayesian optimization method, d) a particle swarm optimization method, e) a genetic method, f) a grid search method, g) a dynamic search method, h) other search method, or i) any combination of a)-h).
62. The non-transitory computer-readable medium of any of Claims 58-61, wherein the plurality of candidate ML models includes a) an unsupervised ML model, b) a supervised ML model, c) a regression ML model, d) a classification ML model, e) a clustering ML model, or f) any combination of a)-e).
63. The non-transitory computer-readable medium of any of Claims 58-62, wherein, to iteratively evaluate each candidate ML model of a plurality of candidate ML models, the sequence of instructions, when loaded and executed by the at least one processor, causes the at least one processor to employ at least one metric and wherein a metric of the at least one metric includes a classification metric, a regression metric, or a clustering metric.
64. The non-transitory computer-readable medium of any of Claims 58-63, wherein, to iteratively evaluate each candidate ML model of a plurality of candidate ML models, the sequence of instructions, when loaded and executed by the at least one processor, causes the at least one processor to employ a plurality of metrics with respective weights.- 56 - 5083032. vlDocket No. 1086.210700165. A non-transitory computer-readable medium for automating machine-learning (ML) model hyperparameter selection, the non-transitory computer-readable medium having encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to:receive at least one criterion and a hyperparameter search space; iteratively evaluate hyperparameter values, within the hyperparameter search space received, for hyperparameters of a ML model until i) a result of the iteratively evaluating of a given iteration indicates that hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit, wherein hyperparameter values for evaluating are selected based on learned performance of the ML model and learned computational cost of the evaluating; andthe hyperparameter values of the given iteration are selected optimum hyperparameters for the ML model.
66. A non-transitory computer-readable medium for automating machine-learning (ML) model and ML model hyperparameter selection, the non-transitory computer-readable medium having encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to:receive at least one criterion and a hyperparameter search space; and iteratively evaluate each candidate ML model of a plurality of candidate ML models and hyperparameter values, within the hyperparameter search space received, for hyperparameters of each candidate ML model until: i) a result of the iteratively evaluating of a given iteration indicates that the candidate ML model and hyperparameter values of the given iteration meet the at least one criterion and ii) evaluating the candidate ML model and hyperparameter values of an iteration subsequent to the given iteration exceeds an evaluation limit, whereincandidate ML models and hyperparameter values for evaluating are selected based on learned performance of the candidate ML models and learned computational cost of the evaluating, and- 57 - 5083032. vlDocket No. 1086.2107001the candidate ML model of the given iteration is a selected optimum ML model and the hyperparameter values of the given iteration are selected optimum hyperparameter values.
67. A non-transitory computer-readable medium for automating machine-learning (ML) model training data selection, the non-transitory computer-readable medium having encoded thereon a sequence of instructions which, when loaded and executed by at least one processor, causes the at least one processor to:receive a time-series dataset for monitoring performance of an asset of a real- world system;produce a reduced time-series dataset by excluding a portion of data of the time-series dataset received;iteratively process the reduced time-series dataset to identify at least one data distribution in the reduced time-series dataset; andautomatically select a given data distribution from the at least one data distribution identified, wherein the given data distribution selected is selected as training data for a ML model and the given data distribution is selected automatically based on a temporal correspondence between the given data distribution selected and a target timeframe for monitoring performance of the asset with the ML model.
68. The non-transitory computer-readable medium of Claim 67, wherein, to select the given data distribution, the sequence of instructions, when loaded and executed by the at least one processor, causes the at least one processor to:identify a target behavior in the given data distribution; andautomatically select the given data distribution based on a) the temporal correspondence between the given data distribution selected and the target timeframe and b) the identifying the target behavior in the given data distribution.
69. The computer-implemented of Claim 68, wherein the target behavior includes a) at least one anomaly, b) at least one failure signature, or c) any combination of a) and b).
70. The non-transitory computer-readable medium of any of Claims 67-69, wherein the sequence of instructions, when loaded and executed by the at least one processor, further causes the at least one processor to:- 58 - 5083032. vlDocket No. 1086.2107001train the ML model using the given data distribution selected automatically, the training producing a trained ML model; anddeploy the trained ML model in the target timeframe.
71. The non-transitory computer-readable medium of any of Claims 67-70, wherein the portion of data excluded corresponds to a) a time period for which the asset was known to be offline, b) a time period for which the asset was associated with a confirmed failure event, or c) any combination of a) and b).
72. The non-transitory computer-readable medium of any of Claims 67-71, wherein the time-series dataset received is sensor data received from at least one sensor coupled to the asset.
73. The non-transitory computer-readable medium of any of Claims 67-72, wherein the time-series dataset includes non- stationary multivariate time series data or stationary multivariate time series data.
74. The non-transitory computer-readable medium of any of Claims 67-73, wherein the given data distribution is selected based on the temporal correspondence and a determination that the given data distribution, in its entirety, encompasses a training data portion and a test data portion in a single distribution.
75. The non-transitory computer-readable medium of any of Claims 67-74, wherein, to iteratively process the reduced time-series dataset to identify the at least one data distribution in the reduced time-series dataset, the sequence of instructions, when loaded and executed by the at least one processor, causes the at least one processor to:determine whether the reduced time-series dataset, in its entirety, corresponds to one distribution and, responsive to determining that the reduced time-series dataset, in its entirety, does not correspond to one distribution, the iteratively processing further includes:iteratively i) dividing the reduced time-series dataset into a plurality of datasets, the iteratively dividing including dividing each dataset of the plurality of datasets into a candidate training data portion and test data portion and ii) evaluating whether each dataset of the plurality of datasets corresponds to one distribution, until a) each dataset of the plurality of datasets corresponds- 59 - 5083032. vlDocket No. 1086.2107001to a respective one distribution or b) a total number of maximum iterations has been performed, wherein the at least one data distribution identified is composed of each candidate training data portion.5083032. vl
