Systems and methods for cooktop event inference

Abstract

Systems and methods are disclosed for predicting unsafe cooktop events through machine-learning analysis of sensor-derived burner image sequences. A cooktop monitoring system collects thermal or multi-modal image data depicting one or more heating elements over time. A dataset assembly process identifies timepoints within each burner sequence and constructs corresponding training instances, each comprising a window of past burner images and a label indicating a cooktop condition at a future horizon time. These instances are used to train a machine-learning model to anticipate conditions such as fire, boil-over, overheating, or unsafe operating status. During deployment, a controller maintains a sliding window of recent burner images and generates predictive classifications or temperature estimates using the trained model. Based on predicted future states, the system initiates actions, including visual or auditory alerts, activation of a fire alarm, or power disconnection. The approach enables proactive mitigation of hazardous cooktop events through real-time predictive inference.

Description

SYSTEMS AND METHODS FOR COOKTOP EVENT INFERENCE FIELD

[0001] The field of the invention relates to systems and methods for monitoring cooktops. More specifically, system and methods for predicting cooktop events, such as fire or other unsafe cooktop events.INTRODUCTION

[0002] Cooking is a leading cause of home fires and home fire injuries. Incidents caused by unsafe cooktop use can lead to injury, property loss, and death. For example, cooktop fires can be caused by leaving the kitchen unattended while cooking, which can be a frequent problem for elderly individuals, individuals with mobility issues, distracted individuals or individuals otherwise prone to forgetfulness. Such individuals are also frequently the most vulnerable when such incidents occur as they may be unable to ameliorate the immediate issue before it escalates into a significant safety risk.

[0003] This can be especially concerning in residential settings, including high-density residential settings. In such environments, fires can quickly propagate to multiple residential units if not properly managed at the source, resulting in risk for large numbers of individuals. Additionally, unlike in commercial settings, building-wide fire protection systems may not be adequate to minimize property damage or injury. Insurers are especially weary of these risks, and in some cases, may decline to provide insurance coverage to property owners in similar high risk settings.

[0004] Existing systems for detecting unsafe cooktop use may use simple detection methods that only take action or provide alerts after a fire has started. Systems that aim to take pre-emptive action using simple sensors and algorithms may run an unduly high risk of incorrectly predicting cooktop events which can lead to false alarms and / or unwanted discharge of fire retardant.SUMMARY

[0005] The following introduction is provided to introduce the reader to the more detailed discussion to follow. The introduction is not intended to limit or define any claimed or as yet unclaimed invention. One or more inventions may reside in any combination orsub-combination of the elements or process steps disclosed in any part of this document including its claims and figures. The described systems and methods provide a predictive training dataset, ML model and inference system using horizon-based future state labels, enabling proactive and preventive safety actions and risk mitigation.

[0006] The systems and methods disclosed in the specification relate to systems and methods for predicting hazardous cooktop events before they occur. Using thermal and other sensor-derived image data, the technology captures time-ordered views of individual burners and converts this information into structured training datasets. These datasets are composed of “timepoint training instances” that pair a window of past burner images with a label representing the state of the burner at a future horizon time. This window-and-horizon structure enables machine-learning models to learn not only what a hazardous condition looks like, but how it develops over time.

[0007] To generate the training data, the system acquires full cooktop images, segments them into burner-specific sequences, and applies optional preprocessing such as rotation, normalization, de-skewing, and noise augmentation. Labelling may done manually or may be automated and supports a variety of conditions including fire, boil-over, overheating, and uncovered burners. These dataset assembly techniques produce rich, temporally aligned examples suitable fortraining predictive models.

[0008] The describe embodiments further provide methods and systems for training machine-learning models using the structured dataset. The models (such as CNNs, LSTMs, transformers, or hybrid architectures) consume the input windows and learn to infer future burner states at the horizon time. The training process includes parameter optimization, hyperparameter tuning, dataset partitioning, augmentation strategies, and evaluation on a separate test set. This ensures robust predictive capability across diverse cooktop configurations and cooking scenarios.

[0009] At runtime, a monitoring device equipped with sensors and a controller continually collects burner images and maintains a sliding input window. The trained prediction model processes these windows to forecast hazardous or undesirable cooktop events before they occur. If a cooktop event is predicted, the system can initiate responsive actions such as visual or auditory warnings, remote alerts, activation of fire safety equipment, or automated disconnection of power to the cooktop or a specific heating element.

[0010] Together, these aspects provide a comprehensive framework for proactive cooktop safety. By combining intelligent data assembly, predictive machine-learning modeling, real-time inference, and automated safety interventions, the technology significantly reduces the likelihood of fires and other dangerous cooking events. This predictive approach addresses a longstanding gap in traditional cooktop safety systems, which typically respond only after unsafe conditions have already emerged.

[0011] In a broad aspect, in accordance with one or more embodiments, there is provided herein a method for assembling a cooktop event prediction training dataset. The method comprises providing a burner image sequence, the burner image sequence comprising a plurality of burner thermal images, identifying, at a processor, a plurality of timepoints in the burner image sequence, and for each timepoint in the burner image sequence, assembling, at the processor, a timepoint training instance. The timepoint training instance comprises a set of burner thermal images corresponding to a window time period preceding (or otherwise corresponding to) the timepoint, and a cooktop event label corresponding to a state of a condition of the burner at a horizon time corresponding to the timepoint.

[0012] In some embodiments, the burner image sequence is prepared by: imaging a whole cooktop, thereby generating a cooktop image sequence; segmenting the cooktop image sequence into one or more burner image sequences; and selecting the burner image sequence from the one or more burner image sequences.

[0013] In some embodiments, the burner image sequence is prepared by performing at least one of the following transformations on the burner image sequence: a normalization, a flip, and a de-skew.

[0014] In some embodiments, the burner image sequence is prepared by adding noise to the burner image sequence.

[0015] In some embodiments, the added noise is one or more types of noise selected from the group consisting of: Gaussian noise, Gaussian blur noise, salt-and-pepper noise, Poisson noise, speckle noise, motion blur noise and color jitter noise.

[0016] In some embodiments, at least a portion of the burner thermal images are labelled corresponding to the state of the condition of the burner depicted in the frame.

[0017] In some embodiments, the burner image sequence is prepared by performing at least one rotation.

[0018] In some embodiments, the plurality of burner thermal images are extracted from a cooktop video.

[0019] In some embodiments, the plurality of burner frames are captured at a constant frame rate.

[0020] In some embodiments, the timepoints are selected at a timepoint frequency, the timepoint frequency being equal to the frame rate.

[0021] In some embodiments, the timepoints are selected at a timepoint frequency, the timepoint frequency being less than the frame rate.

[0022] In some embodiments, the conditions comprise at least one of: fire, boil-over, operating status, pot-too-hot, and uncovered burner.

[0023] In some embodiments, at least a portion of the cooktop event labels is received from a human performing manual labelling on the cooktop image sequence.

[0024] In some embodiments, at least a portion of the cooktop event labels is received from a computing system performing automated labelling of the cooktop image sequence using computer-vision-based techniques.

[0025] In some embodiments, the cooktop image sequence is an element within a cooktop image sequence dataset, the cooktop image dataset comprising a plurality of elements.

[0026] In some embodiments, the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

[0027] In some embodiments, the average value is based on the cooktop image sequence dataset.

[0028] In another broad aspect, in accordance with one or more embodiments, there is provided herein a system for assembling a cooktop event prediction training dataset from a plurality of cooktop prediction data series. The system comprises a memory, a processor in communication with the memory, the processor configured to provide a burner image sequence, the burner image sequence comprising a plurality of burner thermal images, identify a plurality of timepoints in the burner image sequence, and for each timepoint in the burner image sequence, assembling a timepoint training instance, the timepoint training instance comprising a set of burner thermal images corresponding to a window time period corresponding to the timepoint, and a cooktopevent label corresponding to a state of a condition of the burner at a horizon time corresponding to the timepoint.

[0029] In some embodiments, the burner image sequence is prepared by: imaging a whole cooktop, thereby generating a cooktop image sequence; segmenting the cooktop image sequence into one or more burner image sequences; and selecting the burner image sequence from the one or more burner image sequences.

[0030] In some embodiments, the burner image sequence is prepared by performing at least one of the following transformations on the burner image sequence: a normalization, a flip, and a de-skew.

[0031] In some embodiments, the burner image sequence is prepared by adding noise to the burner image sequence.

[0032] In some embodiments, the added noise is one or more types of noise selected from the group consisting of: Gaussian noise, Gaussian blur noise, salt-and-pepper noise, Poisson noise, speckle noise, motion blur noise and color jitter noise.

[0033] In some embodiments, at least a portion of the burner thermal images are labelled corresponding to the state of the condition of the burner depicted in the frame.

[0034] In some embodiments, the burner image sequence is prepared by performing at least one rotation.

[0035] In some embodiments, the plurality of burner thermal images are extracted from a cooktop video.

[0036] The system of claim 19 wherein the plurality of burner frames are captured at a constant frame rate.

[0037] In some embodiments, the timepoints are selected at a timepoint frequency, the timepoint frequency being equal to the frame rate.

[0038] In some embodiments, the timepoints are selected at a timepoint frequency, the timepoint frequency being less than the frame rate.

[0039] In some embodiments, the conditions comprise at least one of: fire, boil-over, operating status, pot-too-hot, and uncovered burner.

[0040] In some embodiments, at least a portion of the cooktop event labels is received from a human performing manual labelling on the cooktop image sequence.

[0041] In some embodiments, at least a portion of the cooktop event labels is received from a computing system performing automated labelling of the cooktop image sequence using computer-vision-based techniques.

[0042] In some embodiments, the cooktop image sequence is an element within a cooktop image sequence dataset, the cooktop image dataset comprising a plurality of elements.

[0043] In some embodiments, the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

[0044] In some embodiments, the average value is based on the cooktop image sequence dataset.

[0045] In another broad aspect, in accordance with one or more embodiments, there is provided herein a method for predicting a cooktop event at a cooktop. The method comprises collecting, at a sensor, a burner image sequence comprising a plurality of burner thermal images, generating, at a controller, a cooktop event classification based on the plurality of burner thermal images using a cooktop event prediction model, the cooktop event classification corresponding to a predicted state of a condition of the cooktop corresponding to a horizon time after the burner image sequence, initiating, at the controller, a warning action based on the cooktop event classification.

[0046] In some embodiments, the warning action comprises one or more of: an auditory notification and a visual notification.

[0047] In some embodiments, the warning action comprises activating a fire alarm system.

[0048] In some embodiments, the burner image sequence is prepared by: imaging the cooktop, thereby generating a cooktop image sequence; segmenting the cooktop image sequence into one or more burner image sequences; and selecting the burner image sequence from the one or more burner image sequences.

[0049] In some embodiments, the burner image sequence is prepared by performing at least one of the following transformations on the burner image sequence: a normalization, a flip, and a de-skew.

[0050] In some embodiments, the burner image sequence is prepared by adding noise to the burner image sequence.

[0051] In some embodiments, the added noise is one or more types of noise selected from the group consisting of: Gaussian noise, Gaussian blur noise, salt-and-pepper noise, Poisson noise, speckle noise, motion blur noise and color jitter noise.

[0052] In some embodiments, the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

[0053] In some embodiments, the burner image sequence is prepared by performing at least one rotation.

[0054] In some embodiments, the plurality of burner thermal images are extracted from a cooktop video.

[0055] In some embodiments, the sensor comprises a thermal imaging sensor.

[0056] In some embodiments, the conditions comprise at least one of: fire, boil-over, operating status, pot-too-hot, and uncovered burner.

[0057] In another broad aspect, in accordance with one or more embodiments, there is provided a system for predicting a cooktop event at a cooktop, the system comprising: a sensor, configured to collect a burner image sequence comprising a plurality of burner thermal images; a memory unit, having stored thereon: a cooktop event prediction model; a controller, configured to: generate, using the cooktop event prediction model, a cooktop event label based on the plurality of burner thermal images, wherein the cooktop event label corresponds to a predicted state of a condition of the cooktop corresponding to a horizon time after the burner image sequence; and initiate a warning action based on the cooktop event label.

[0058] In some embodiments, the warning action comprises one or more of: an auditory notification and a visual notification.

[0059] In some embodiments, the warning action comprises activating a fire alarm system.

[0060] In some embodiments, the burner image sequence is prepared by: imaging the cooktop, thereby generating a cooktop image sequence; segmenting the cooktop image sequence into one or more burner image sequences; and selecting the burner image sequence from the one or more burner image sequences.

[0061] In some embodiments, the burner image sequence is prepared by performing at least one of the following transformations on the burner image sequence: a normalization, a flip, and a de-skew.

[0062] In some embodiments, the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

[0063] In some embodiments, burner image sequence is prepared by performing at least one rotation.

[0064] In some embodiments, the plurality of burner thermal images are extracted from a cooktop video.

[0065] In some embodiments, the sensor comprises a thermal imaging sensor.

[0066] In some embodiments, the conditions comprise at least one of: fire, boil-over, operating status, pot-too-hot, and uncovered burner.

[0067] In another broad aspect, in accordance with one or more embodiments, there is provide a method for generating a cooktop event prediction model for predicting a cooktop event, comprising: providing, at a memory, a cooktop event prediction training dataset, the cooktop event prediction training dataset comprising a plurality of timepoint training instances, each timepoint training instance comprising: a set of training burner thermal images corresponding to a window time period; and a training cooktop event label corresponding to a state of a condition of a training burner at a horizon time after the window time period; operating, at the processor, a machine learning model to produce one or more cooktop event labels based on one or more sets of training burner thermal images corresponding to one or more timepoint training instances; and modifying, at the processor, one or more parameters of the machine learning model based on a comparison between the one or more inferred cooktop event labels and one or more training cooktop event labels corresponding to the one or more timepoint training instances.

[0068] In some embodiments, the method further comprises: dividing, at the processor, a portion of the cooktop event prediction training dataset into a model training dataset, model validation dataset, and model testing dataset.

[0069] In some embodiments, the modifying the one or more parameters of the machine learning model further comprises: modifying, at the processor, one or more model weights of the machine learning model based on the model training dataset; and modifying, at the processor, one or more hyperparameters associated with machine learning model based on the model validation dataset.

[0070] In some embodiments, the method further comprises testing, at the processor, the machine learning model based on the testing dataset once a training error has reached a threshold.

[0071] In some embodiments, the set of training burner thermal images are extracted from a cooktop video.

[0072] In some embodiments, the training burner image sequence is preprocessed by performing at least one of the following transformations on the training burner image sequence: a normalization, a flip, and a de-skew.

[0073] In some embodiments, the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

[0074] In some embodiments, the training burner image sequence is prepared by performing at least one rotation.

[0075] In another broad aspect, in accordance with one or more embodiments, there is provided a system for generating a cooktop event prediction model for predicting a cooktop event. The system comprises a memory, having stored thereon a cooktop event prediction training dataset, the cooktop event prediction training dataset comprising a plurality of timepoint training instances, each timepoint training instance comprising a set of training burner thermal images corresponding to a window time period, and a training cooktop event label corresponding to a state of a condition of a training burner at a horizon time after the window time period, a processor, configured to produce, using a machine learning model, one or more cooktop event labels based on one or more sets of training burner thermal images corresponding to one or more timepoint training instances, and modify one or more parameters of the machine learning model based on a comparison between the one or more inferred cooktop event labels and one or more training cooktop event labels corresponding to the one or more timepoint training instances.

[0076] In some embodiments, the processor is further configured to: divide a portion of the cooktop event prediction training dataset into a model training dataset, model validation dataset, and model testing dataset.

[0077] In some embodiments, the modifying the one or more parameters of the machine learning model further comprises: modifying, at the processor, one or more model weights of the machine learning model based on the model training dataset; and modifying, at the processor, one or more hyperparameters associated with machine learning model based on the model validation dataset.

[0078] In some embodiments, the processor is further configured to test, at the processor, the machine learning model based on the testing dataset once a training error has reached a threshold.

[0079] In some embodiments, the set of training burner thermal images are extracted from a cooktop video.

[0080] In some embodiments, the training burner image sequence is preprocessed by performing at least one of the following transformations on the training burner image sequence: a normalization, a flip, and a de-skew.

[0081] In some embodiments, the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

[0082] In some embodiments, the training burner image sequence is prepared by performing at least one rotation.

[0083] In another broad aspect, in accordance with one or more embodiments, there is provided a method for assembling a cooktop-event prediction training dataset, the method comprising: providing a burner image sequence comprising a plurality of time-ordered burner images derived from sensor data; identifying, by a processor, a plurality of timepoints within the burner image sequence; and for each timepoint, assembling a timepoint training instance comprising: (i) an input window including a sequence of burner images associated with a window time period defined relative to the timepoint; and (ii) a cooktop-event label corresponding to a state of a condition at a horizon time defined relative to the timepoint.

[0084] In some embodiments, the burner image sequence is generated by: imaging an entire cooktop to obtain a cooktop image sequence; segmenting the cooktop imagesequence into one or more burner-specific image sequences; and selecting one of the burner-specific image sequences as the burner image sequence.

[0085] In some embodiments, the burner images comprise thermal images, visible-light images, infrared images, or a combination thereof obtained from one or more sensors.

[0086] In some embodiments, the method further comprises performing one or more transformations on the burner image sequence selected from: normalization, flipping, rotation, and de-skewing.

[0087] In some embodiments, the normalization comprises scaling pixel values based on an average value, the average value computed per image, per sequence, or over a dataset of sequences.

[0088] In some embodiments, the rotation is 90°, 180°, or 270°.

[0089] In some embodiments, the method further comprises adding noise to the burner image sequence, the noise comprising one or more of: Gaussian noise, Gaussian blur, salt-and-pepper, Poisson noise, speckle noise, motion blur, and color jitter.

[0090] In some embodiments, at least a portion of the cooktop-event labels are generated by manual labeling, automated labeling using computer-vision techniques, or a combination thereof.

[0091] In some embodiments, the burner images are frames extracted from a video captured at a source frame rate, and wherein the burner image sequence is sub-sampled from the source frame rate.

[0092] In some embodiments, the timepoints are selected at a timepoint frequency equal to the burner image sequence frame rate.

[0093] In some embodiments, the timepoints are selected at a timepoint frequency less than the burner image sequence frame rate.

[0094] In some embodiments, the condition comprises at least one of: fire, boil-over, operating status, pot-too-hot, uncovered burner, or a temperature value.

[0095] In some embodiments, input window comprises a sliding window of the most recent N frames stored in a buffer.

[0096] In some embodiments, segmenting comprises perspective correction or homography-based de-skewing prior to isolating burner regions.

[0097] In some embodiments, the cooktop-event label is binary, multi-class, multi-label, ora regression value.

[0098] In some embodiments, the method further comprises: creating timepoint training instances that use images for a plurality of burners concurrently, the input window including frames from two or more burner regions.

[0099] In some embodiments, the horizon time is constrained to occur after a last frame of the input window.

[0100] In some embodiments, the method further comprises storing the assembled timepoint training instances in a training dataset file or database structure indexed by sequence identifier, timepoint, window start, window end, and horizon.

[0101] In some embodiments, the sensor data further comprises at least one of: depth data, humidity, ambient temperature, utility draw, audio, or electromagnetic emissions, and the input window includes corresponding time-aligned features.

[0102] In some embodiments, normalization parameters are computed at the dataset level and applied consistently across all sequences.

[0103] In another broad aspect, in accordance with one or more embodiments, there is provided a method for generating a cooktop-event prediction model, the method comprising: providing a training dataset comprising a plurality of timepoint training instances, each instance including an input window of burner images and a corresponding cooktop-event label at a horizon time; operating a machine learning model to produce one or more predicted labels from the input windows; and modifying one or more model parameters based on a comparison between the predicted labels and the corresponding cooktop-event labels.

[0104] In some embodiments, the method further comprises partitioning the training dataset into a model-training subset, a validation subset, and a testing subset.

[0105] In some embodiments, modifying the model parameters comprises updating model weights using a gradient-based optimizer.

[0106] In some embodiments, the method further comprises tuning one or more hyperparameters based on performance on the validation subset.

[0107] In some embodiments, the method further comprises evaluating the trained model on the testing subset after a training error satisfies a stopping criterion.

[0108] In some embodiments, the machine learning model comprises at least one of: a convolutional neural network, a recurrent neural network, a long short-term memory network, a transformer, a 3D-CNN, a support vector machine, a decision tree, a random forest, or an autoencoder.

[0109] In some embodiments, input window is encoded into per-frame feature embeddings which are aggregated temporally by a sequence model.

[0110] In some embodiments, the cooktop-event label comprises a temperature value predicted at the horizon time, and the modifying comprises minimizing a regression loss.

[0111] In some embodiments, the input window includes images from two or more burners and the machine learning model learns cross-burner correlations.

[0112] In some embodiments, at least a portion of the training dataset is augmented using transformations comprising normalization, flipping, rotation, de-skewing, and noise injection.

[0113] In another broad aspect, in accordance with one or more embodiments, there is provided a method for predicting a cooktop event at a cooktop, the method comprising: collecting, by at least one sensor, a burner image sequence comprising a plurality of time-ordered burner images; maintaining, by a controller, an input window comprising a most-recent sequence of burner images; generating, by the controller using a trained cooktop-event prediction model, a cooktop-event classification or regression output corresponding to a predicted state of a condition at a horizon time; and initiating, by the controller, an action based on the generated output.

[0114] In some embodiments, the action comprises issuing a visual notification, an auditory notification, transmitting an alert to a fire alarm system, or disconnecting power to the cooktop or a heating element.

[0115] In some embodiments, the method further comprises escalating the action in stages based on persistence or severity of the predicted condition.

[0116] In some embodiments, the method further comprises imaging the cooktop to obtain a cooktop image sequence, segmenting the cooktop image sequence into oneor more burner-specific image sequences, and selecting a burner-specific image sequence as the burner image sequence.

[0117] In some embodiments, the trained model resides on the controller, in a remote computing resource accessible over a network, or is partitioned between the controller and the remote computing resource.

[0118] In some embodiments, the condition includes at least one of: fire, boil-over, operating status, pot-too-hot, uncovered burner, ora predicted temperature at the horizon time.

[0119] In some embodiments, initiating the action comprises comparing the generated output with a threshold that depends on an operating mode of the cooktop.

[0120] In some embodiments, the method further comprises reconnecting power after the generated output indicates that a predicted unsafe state is no longer present.

[0121] In some embodiments, the burner images are captured at a constant frame rate and the input window advances at a timepoint frequency equal to or less than the frame rate.

[0122] In some embodiments, the method further comprises normalizing the burner images prior to generating the output using dataset-level normalization parameters.

[0123] In another broad aspect, in accordance with one or more embodiments, there is provided a system for assembling a cooktop-event prediction training dataset, comprising: a memory storing sensor-derived cooktop image data; and a processor configured to: provide a burner image sequence comprising a plurality of time-ordered burner images; identify a plurality of timepoints in the burner image sequence; and for each timepoint, assemble a timepoint training instance comprising an input window including burner images associated with a window time period and a cooktop-event label corresponding to a condition at a horizon time.

[0124] In some embodiments, the processor is further configured to perform at least one of: normalization, flipping, rotation, de-skewing, and noise injection on the burner image sequence.

[0125] In some embodiments, the memory further stores labels generated by manual labeling, automated labeling, or both.

[0126] In some embodiments, the burner images comprise thermal images, visible-light images, infrared images, ora combination thereof.

[0127] In another broad aspect, in accordance with one or more embodiments, there is provided a system for generating a cooktop-event prediction model, comprising:a memory storing a training dataset comprising a plurality of timepoint training instances, each instance including an input window and a corresponding cooktop-event label; and a processor configured to: operate a machine learning model to produce one or more cooktop-event labels based on input windows; and modify parameters of the machine learning model based on a comparison between the predicted labels and the corresponding cooktop-event labels.

[0128] In some embodiments, the processor is further configured to partition the training dataset into training, validation, and testing subsets and to tune hyperparameters based on the validation subset.

[0129] In some embodiments, the machine learning model comprises a convolutional neural network, a recurrent neural network, a long short-term memory network, a transformer, a 3D-CNN, a support vector machine, a decision tree, a random forest, or an autoencoder.

[0130] In another broad aspect, in accordance with one or more embodiments, there is provided a system for predicting a cooktop event at a cooktop, comprising:at least one sensor configured to collect a burner image sequence comprising a plurality of time-ordered burner images; a memory storing a trained cooktop-event prediction model; and a controller configured to: maintain an input window comprising a most-recent sequence of burner images; generate, using the trained model, a cooktop-event classification or regression output corresponding to a horizon time; and initiate an action based on the generated output.

[0131] In some embodiments, the at least one sensor comprises a thermal imaging sensor and a visible-light imaging sensor.

[0132] In some embodiments, the action comprises issuing a visual or auditory notification, activating a fire alarm system, or disconnecting power to the cooktop or a heating element.

[0133] In some embodiments, the controller is further configured to image the cooktop, segment the cooktop image sequence into burner-specific image sequences, and select a burner-specific image sequence as the burner image sequence.

[0134] In some embodiments, the trained model is executed on the controller, on a remote computing resource, or is split between the controller and the remote computing resource.

[0135] In another broad aspect, in accordance with one or more embodiments, there is provided a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising the methods described above.

[0136] In some embodiments, the window time period corresponds to a time period of up to 60 seconds and the horizon time corresponds to a time period of up to 120 seconds.

[0137] In some embodiments, the window time period corresponds to a time period of between two to ten seconds and the horizon time corresponds to a time period of between five to sixty seconds.

[0138] In some embodiments, the segmentation employs a fixed spatial mapping to known burner positions.

[0139] In some embodiments, the regression output is a predicted maximum burner-surface temperature over the horizon time and the action is triggered when the predicted maximum exceeds a threshold.

[0140] In some embodiments, the controller escalates from a visual notification to an auditory notification and then to power disconnection when the predicted probability of fire remains above a threshold for a first time interval and a second, longer time interval, respectively.DRAWINGS

[0141] Several embodiments will now be described in detail with reference to the drawings, in which:FIG. 1 shows a schematic diagram of an example environment in which a system for predicting a cooktop event may be used.FIG. 2 shows a block diagram of an example cooktop monitoring unit.FIG. 3 shows a schematic diagram of an example system for collecting data. FIG. 4 shows a block diagram of an example system data assembly device.FIG. 5 shows a flowchart of an example method for assembling a cooktop event prediction training dataset.FIG. 6 shows an example mounting configurations of an example monitoring unit. FIG. 7 shows an schematic diagram of an example assembly of burner image sequences for a training dataset.FIG. 8 shows an example method for assembling a cooktop event prediction dataset.FIG. 9 shows an example method for generating a cooktop event prediction model. FIG. 10 shows an example method for predicting a cooktop event.FIG. 11 shows an example method for using a model to generate a cooktop event classification.FIG. 12 shows a block diagram of an example computing device.

[0142] The drawings, described below, are provided for purposes of illustration, and not of limitation, of the aspects and features of various examples of embodiments described herein. For simplicity and clarity of illustration, elements shown in the drawings have not necessarily been drawn to scale. The dimensions of some of the elements may be exaggerated relative to other elements for clarity. It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the drawings to indicate corresponding or analogous elements or steps.DESCRIPTION OF VARIOUS EMBODIMENTS

[0143] Various embodiments in accordance with the teachings herein will be described below to provide an example of at least one embodiment of the claimed subject matter. No embodiment described herein limits any claimed subject matter. The claimed subject matter is not limited to devices, systems or methods having all of the features of any one of the devices, systems or methods described below or to features common tomultiple or all of the devices, systems or methods described herein. It is possible that there may be a device, system or method described herein that is not an embodiment of any claimed subject matter. Any subject matter that is described herein that is not claimed in this document may be the subject matter of another protective instrument, for example, a continuing patent application, and the applicants, inventors or owners do not intend to abandon, disclaim or dedicate to the public any such subject matter by its disclosure in this document.

[0144] For simplicity and clarity of illustration, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the subject matter described herein. However, it will be understood by those of ordinary skill in the art that the subject matter described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the subject matter described herein. The description is not to be considered as limiting the scope of the subject matter described herein.

[0145] It should also be noted that the terms “coupled” or “coupling” as used herein can have several different meanings depending in the context in which these terms are used. For example, the terms coupled or coupling can have a mechanical, fluidic or electrical connotation. For example, as used herein, the terms coupled or coupling can indicate that two elements or devices can be directly connected to one another or connected to one another through one or more intermediate elements or devices via an electrical or magnetic signal, electrical connection, an electrical element or a mechanical element depending on the particular context. Furthermore, coupled electrical elements may send and / or receive data.

[0146] Unless the context requires otherwise, throughout the specification and claims which follow, the word “comprise” and variations thereof, such as, “comprises” and “comprising” are to be construed in an open, inclusive sense, that is, as “including, but not limited to”.

[0147] It should also be noted that, as used herein, the wording “and / or” is intended to represent an inclusive-or. That is, “X and / or Y” is intended to mean X or Y or both, for example. As a further example, “X, Y, and / or Z” is intended to mean X or Y or Z or any combination thereof.

[0148] It should be noted that terms of degree such as "substantially", "about" and "approximately" as used herein mean a reasonable amount of deviation of the modified term such that the end result is not significantly changed. These terms of degree may also be construed as including a deviation of the modified term, such as by 1%, 2%, 5% or 10%, for example, if this deviation does not negate the meaning of the term it modifies.

[0149] Furthermore, the recitation of numerical ranges by endpoints herein includes all numbers and fractions subsumed within that range (e.g. 1 to 5 includes 1 , 1.5, 2, 2.75, 3, 3.90, 4, and 5). It is also to be understood that all numbers and fractions thereof are presumed to be modified by the term "about" which means a variation of up to a certain amount of the number to which reference is being made if the end result is not significantly changed, such as 1%, 2%, 5%, or 10%, for example.

[0150] Reference throughout this specification to “one embodiment”, “an embodiment”, “at least one embodiment” or “some embodiments” means that one or more particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, unless otherwise specified to be not combinable or to be alternative options.

[0151] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the content clearly dictates otherwise. It should also be noted that the term “or” is generally employed in its broadest sense, that is, as meaning “and / or” unless the content clearly dictates otherwise.

[0152] Similarly, throughout this specification and the appended claims the term “communicative” as in “communicative pathway,” “communicative coupling,” and in variants such as “communicatively coupled,” is generally used to refer to any engineered arrangement for transferring and / or exchanging information. Exemplary communicative pathways include, but are not limited to, electrically conductive pathways (e.g., electrically conductive wires, electrically conductive traces), magnetic pathways (e.g., magnetic media), optical pathways (e.g., optical fiber), electromagnetically radiative pathways (e.g., radio waves), or any combination thereof. Exemplary communicative couplings include, but are not limited to, electrical couplings, magnetic couplings, optical couplings, radio couplings, or any combination thereof.

[0153] Throughout this specification and the appended claims, infinitive verb forms are often used. Examples include, without limitation: “to detect,” “to provide,” “to transmit,”“to communicate,” “to process,” “to route,” and the like. Unless the specific context requires otherwise, such infinitive verb forms are used in an open, inclusive sense, that is as “to, at least, detect,” to, at least, provide,” “to, at least, transmit,” and so on.

[0154] The example systems and methods described herein may be implemented as a combination of hardware or software. In some cases, the examples described herein may be implemented, at least in part, by using one or more computer programs, executing on one or more programmable devices comprising at least one processing element, and a data storage element (including volatile memory, non-volatile memory, storage elements, or any combination thereof). These devices may also have at least one input device (e.g. a keyboard, mouse, touchscreen, or the like), and at least one output device (e.g. a display screen, a printer, a wireless radio, or the like) depending on the nature of the device.

[0155] Some elements that are used to implement at least part of the systems, methods, and devices described herein may be implemented via software that is written in a high-level procedural language such as object-oriented programming. The program code may be written in C++, C#, JavaScript, Python, or any other suitable programming language and may comprise modules or classes, as is known to those skilled in object-oriented programming. Alternatively, or in addition thereto, some of these elements implemented via software may be written in assembly language, machine language, or firmware as needed. In either case, the language may be a compiled or interpreted language.

[0156] At least some of these software programs may be stored on a computer readable medium such as, but not limited to, a ROM, a magnetic disk, an optical disc, a USB key, and the like that is readable by a device having at least one processor, an operating system, and the associated hardware and software that is used to implement the functionality of at least one of the methods described herein. The software program code, when read by the device, configures the device to operate in a new, specific, and predefined manner (e.g., as a specific-purpose computer) in order to perform at least one of the methods described herein.

[0157] Furthermore, at least some of the programs associated with the systems and methods described herein may be capable of being distributed in a computer program product including a computer readable medium that bears computer usable instructions for one or more processors. The medium may be provided in various forms, includingnon-transitory forms such as, but not limited to, one or more diskettes, compact disks, tapes, chips, and magnetic and electronic storage. Alternatively, the medium may be transitory in nature such as, but not limited to, wire-line transmissions, satellite transmissions, internet transmissions (e.g. downloads), media, digital and analog signals, and the like. The computer useable instructions may also be in various formats, including compiled and non-compiled code.

[0158] Reference is made to FIG. 1, which shows an example environment 100 in which a system for predicting a cooktop event may be used. A cooktop monitoring unit 102 may monitor a cooktop 104, a surrounding area of the cooktop 104 (e.g., the space in front of the cooktop), and / or a cooktop user 106. Cooktop monitoring unit 102 may operate to predict a future occurrence of a cooktop event on cooktop 104. A cooktop event may be any activity or situation on a cooktop that is or may result in an unsafe use of the cooktop or in an unsafe cooking condition. For example, cooktop events may include, but are not limited to, fire, unattended cooking, high temperature uncovered heating element(s), high temperature cookware, high temperature food contained in cookware, burning food, undercooked food, heating element left on and child operating cooktop. Environment 100 may be, for example, a commercial kitchen, a residential kitchen, or any other setting in which a cooktop may be used.

[0159] Cooktop 104 may be any cooktop known in the art. For example, cooktop 104 may be a gas cooktop, an electric cooktop (radiant or coil), an induction cooktop, or any combination thereof. The cooktop may be a stand-alone unit or may be built into a range (i.e. , a single unit having an oven and a cooktop). Cooktop 104 may contain a power unit 108, a controller 112, one or more heating elements 114, and, in some embodiments, a communications module 110. The cooktop may have any number of heating elements 114. The heating elements 114 may be of any shape and size suitable for heating cookware. The heating elements may or may not be the same shape and size as an adjacent heating element. Each heating element 114 may be used with cookware such as a pot, pan, skillet, or any other item used to cook foods on a cooktop with a heating element.

[0160] Power unit 108 provides power to the various parts of cooktop 104, including, but not limited to, controller 112, communications module 110, and heating elements 114. Power unit 108 may include a power distribution system within cooktop 104 and a power connection to an external power supply such as a 120V or 240V ACsupply. Power unit 108 may contain equipment to facilitate distribution of power throughout cooktop 104 such as busses, cabling, and connectors. Power unit 108 may further contain components such as power regulators, transformers, overcurrent protection, interrupting devices, and more. Power unit 108 may contain internal controls to control various aspects of power delivery to and within the cooktop 104.

[0161] Controller 112 may be any device that controls the operation of the heating elements 114. Controller 112 may control each of the heating elements 114 to reach a desired heating element output in accordance with inputs to the cooktop. For example, inputs may be received from an interface connected to the controller containing such controls as buttons, knobs, and switches. Cooktop 104 may be controlled by a cooktop user 106. Cooktop user 106 may be any user of the cooktop, such as someone using the cooktop 104 to cook food. In some embodiments, cooktop user 106 can include someone standing near the cooktop or monitoring the cooktop, but not necessarily using the cooktop.

[0162] Cooktop surface monitoring unit 102 may capture data relating to the operation of cooktop 104, including data relating to the heating elements 114 of cooktop 104. Cooktop surface monitoring unit 102 may be mounted above cooktop 104 to capture data relating to cooktop 104 from above. For example, unit 104 may be mounted to an overhead exhaust hood unit so as to be positioned directly over the cooktop. As another example, unit 102 may be mounted on onto an adjacent wall from cooktop 104 such that the unit 102 can view the cooktop 104 from above. FIG. 6 shows several example mounting configurations of an example cooktop surface monitoring unit 102 when mounted onto a wall, in this example, behind a cooktop. Example cooktop surface mounting unit 102 is shown at a first position 580 and a second position 582. It should however be noted that cooktop surface monitoring unit 102 can be mounted in any manner that allows unit 102 to capture thermal images of the cooktop 104, items on the cooktop 104 or activity taking place on cooktop 104.

[0163] Reference is next made to FIG. 2, which shows a block diagram of an example cooktop monitoring unit 102. A system for predicting a cooktop event may be implemented using the cooktop surface monitoring unit 102. The example cooktop surface monitoring unit 102 may contain an indicator module 212, a controller 208, a memory 210, sensors 214, power source 202, and a communications module 204. Unit 102 may collect and process data using controller 208.

[0164] Unit 102 may collect and process data using controller 208. The controller 208 may contain any processor suitable for analyzing collected data. The controller 208 may include machine learning capabilities. For example, the controller may be configured to operate a trained machine learning model.

[0165] Sensors 214 may be any sensor or combination of sensors suitable for collecting data relating to the operation of cooktop 104. Sensors 214 may collect data by (a) monitoring inputs and / or outputs from a cooktop (e.g., utility draw such as amount of electricity or gas provided to the cooktop, cooktop surface temperature, etc); (b) monitoring the environment surrounding a cooktop (e.g., presence of a user in an area in front of the cooktop, the humidity and ambient temperature of the cooktop); (c) monitoring mechanical and / or electrical signals within the cooktop (i.e., the sensor(s) may be integrated into the cooktop); and / or (d) recording auditory, visual, or electromagnetic signals emitted by the cooktop (e.g., microphone sounds, recording images or video using visible light or infrared light). For example, sensors 214 could contain a thermal imaging sensor that collects a series of images or records a video of the thermal output of a cooktop.

[0166] Sensors 214 may collect data relating to the entire cooktop, an individual heating element, the cooking implements or utensils used on the cooktop (such as pots, pans, tongs, ladles, spatulas, spoons, knives, etc.), the food on the cooktop (including food within or being moved or manipulated using a utensil), or any other data relating to the operation of the cooktop and the state of items on or near the cooktop.

[0167] Memory 210 may be any form of memory capable of storing, reading, and writing data. For example, memory 254 may include non-volatile memory components such as SSD, eMMC, UFS, XFMD, and / or read-only memory. Memory 210 may also include volatile memory such as RAM and cache memory. Memory 210 may be a combination of multiple different storage components, for example, a combination of volatile and non-volatile memory components. Memory 210 may have stored upon it one or more trained machine learning prediction models 220 for predicting a cooktop event. Memory 210 may also contain data storage 222 for storing data. For example, thermal data relating to a cooktop 104 collected during operation of device 104 may be stored in memory 210.

[0168] Communications module 204 may be capable of sending or receiving data to external systems. Communications module 204 may contain one or any number ofdevices with the functionality of receiving inputs to the device or communicating outputs generated by the device. The communications module 204 may contain hardware configured to communicate wirelessly, for example, through radio-frequency protocols such as Wi-Fi, Bluetooth, ZigBee, 5G, LTE, any other proprietary protocol, or analog radio-frequency transmission techniques. The communications module 258 may also contain hardware configured for wired communication, for example, through digital means such as ethernet, USB, UART, RS-232, RS-485, or through analogue means such as voltage or current signals, such as, for example, 0-10V voltage or 4-20mA current signals.

[0169] For example, communications module 204 may contain a network card configured to communicate wirelessly using IEEE 802.11 protocol. The communications module 204 may then be capable of communicating with smart home appliances such as hubs, controllers, devices, or appliances directly. The communications module 204 may be in direct communication with the cooktop being monitored and may be capable of sending control signals to control the operation of the cooktop or receive data about the current operational status of the cooktop.

[0170] As another example, communications module 204 may be in wired communication with an external fire alarm monitoring system. In such instances, the communications module 204 may contain, for example, one or more output terminals for connection with a building fire alarm notification circuit.

[0171] In some embodiments, part or all of a trained machine learning model used by controller 208 may be stored in a remote location accessible to the controller 208 rather than in memory 210, for example, using cloud computing techniques. In such instances, communications module 204 may operate to send and receive input / output data to and from the remotely stored machine learning model.

[0172] Indicator module 212 may be used to provide visual or auditory notifications. For example, speakers may be used to emit an auditory notification and coloured LED lights may be used for visual notification. The notifications may relate to operational status, for example, an LED light indicating green for normal operation or yellow for errors. The notifications may also include cooktop event notifications, for example, a red LED light indicating unsafe operation of a cooktop.

[0173] Power unit 202 may provide power to the other components of unit 104. Power unit 202 may include all hardware and circuitry for providing, converting, regulating,and / or conditioning power for the power unit 202. For example, power unit 202 may include a power supply for receiving power from an external AC power source. Alternatively, or in addition, power unit 202 may include internal sources of power such as batteries, including lithium ion, alkaline, button cell, etc.

[0174] Reference is next made to FIG. 3, which illustrates an example setup 300 for collecting data. Setup 300 contains a collection device 302, a cooktop 308, and a data assembly device 306. The collection device 302 may be used to collect data relating to cooktop operation and cooktop events from cooktop 308. The data collected from collection device 302 may be transferred to data assembly device 306 to produce a training data set for training a machine learning model to predict cooktop events. In some embodiments, data assembly device 306 may receive data from multiple data collection devices 302, each of which monitors a different cooktop 308.

[0175] Setup 300 may be configured in a manner similar to environment 100 of FIG. 1 so as to capture data similar to that which would be seen during runtime of the resulting trained machine learning model. However, it should be noted that setups may differ greatly across different types of kitchens, facilities, and stove configurations. As such, a greater variety of setups 300 may be used to accommodate for this. However, this is not necessary as, in some embodiments, a lower variety of setups 300 may be used and, instead, the data collected using setup 300 can be pre-processed in ways to simulate a greater amount of variation within the collected data set.

[0176] Collection device 302 may consist of one or more devices for collecting data from cooktops relating to cooktop operation and cooktop events. Collection device 302 contains at least one sensor for collecting time series thermal data relating to cooktop operation. For example, collection device 302 may capture a series of thermal images of the heating elements of cooktop 308. Collection device 302 may also capture data in the form of video, which may later be converted to image data.

[0177] Collection device 302 can be the cooktop monitoring unit 102 of FIGs. 1 and 2B but could also be another device that contains a sensor capable of capturing thermal data. Collection device 302 may contain means to transfer or export the collected data from the collection device 302. For example, wireless means such as WiFi or Bluetooth can be used. Wired means such as USB or Ethernet could also be used in addition or alternatively. As another example, the data may be stored on removable storage medium such as SD card or USB flash drive, and the data may then be transferred after collectionto an external device such as a computer or server. The collected data may be transferred in real-time as collection is occurring, at periodic intervals, or at any time as is desired.

[0178] Collection device 302 may further contain additional sensors for collecting various other data. For example, the additional sensors may include humidity, temperature, motion, sound and other data. Additionally, collection device 302 may contain an RGB imaging sensor for capturing time series RGB data relating to cooktop operation. For example, RGB videos and images may be captured contemporaneously with the thermal data.

[0179] Cooktop 308 consist of one or more cooktop units to be used for data collection. Cooktops 308 may consist of any cooktop known in the art. For example, any one of the cooktops 308 may be a gas cooktop, an electric cooktop (radiant or coil), and induction cooktop, or any combination thereof. The cooktops may be a stand-alone unit or may be built into a range (i.e. , a single unit having an oven and a cooktop).

[0180] Setup 300 may be located in a premise or facility used for data collection. For example, cooktops 308 may be located at a data collection premise containing multiple different types of cooktops. The cooktops may be operated in various typical cooking conditions, such as frying, boiling, and steaming foods, to collect data on typical operating conditions. The cooktops may also be operated in unsafe cooking conditions, for example to simulate various dangerous conditions such as boil overs, overtemperature, and leaving food unattended, to collect data on unsafe cooking operating conditions as well as the occurrence of cooktop events.

[0181] Reference is next made to FIG. 4 in conjunction with FIG. 3. FIG. 4 shows a block diagram of an example data assembly device 306. Data processing device 306 may be used to generate training data for training a machine learning model to predict cooktop events based on the data collected by the data collection units 302. Data assembly device 306 may contain processor 408, memory 410, power unit 402, I / O unit 414, and communications module 404.

[0182] Processor 408 controls the operation of the data assembly device 306. The processor 408 can be any suitable processor, controller or digital signal processor that can provide sufficient processing power depending on the configuration, purposes and requirements of data assembly device 306 as is known by those skilled in the art. For example, the processor 408 may be a high-performance general processor. In alternativeembodiments, the processor unit 408 can include more than one processor with each processor being configured to perform different dedicated tasks. The processor 408 may include a standard processor, such as an Intel® processor or an AMD® processor. Processor 408 may also include graphics processing units such as Nvidia GeForce, AMD Radeon, Nvidia A100, and any other type of graphics processing unit suitable for parallel computing.

[0183] The I / O unit 414 can include at least one of a mouse, a keyboard, a touch screen, a thumbwheel, a trackpad, a trackball, a card-reader, an audio source, a microphone, voice recognition software and the like again depending on the particular implementation of the data assembly device 306. In some cases, some of these components can be integrated with one another.

[0184] The power unit 402 can be any suitable power source that provides power to data assembly device 306 such as a power adaptor or a rechargeable battery pack depending on the implementation of the device 306 as is known by those skilled in the art.

[0185] The memory unit 410 can include RAM, ROM, one or more hard drives, one or more flash drives or some other suitable data storage elements such as disk drives, etc. The memory unit 410 may contain software code for implementing an operating system, programs, and for implementing algorithms for assembling a training dataset for training a machine learning model.

[0186] The memory unit 410 may store collected data set 420, assembled data set 422, and label 424. Collected data set 420 may contain data to be assembled into a training data set. For example, collected data set 420 may contain data relating to cooktop 308 from various data collection devices 302. However, collected data set 420 may contain raw data obtained from any source for assembly into a training data set. The data may be stored in a data storage file format, such as CSV, TSV, Parquet, XML, or any other file format suitable for storing data. The data may be processed or unprocessed. Processing may entail normalizing, removal of outliers, removal of corrupted or erroneous data, or any other number of operations that may improve training data quality for machine learning. Assembled data set 422 may contain data that has been processed and assembled into data suitable for use as training data for machine learning purposes. Labels 424 may contain labels for labelling collected data 420 to produce assembled data 422.

[0187] Reference is next made to FIG. 5, which shows a flowchart of a method 500 for assembling a cooktop event prediction training dataset. The cooktop event prediction training dataset may be used to train a machine learning model. Method 500 may be implemented, for example, on the processor of the data assembly device 306.

[0188] The method begins, at 502, with providing a burner image sequence, the burner image sequence comprising a plurality of burner thermal images. The burner image sequence may be a sequentially (for example, chronologically) ordered sequence of burner thermal images. The burner thermal images may depict, for example, a heating element of a cooktop. For example, the burner image sequence may be captured as a part of testing setup 300, wherein a thermal sensor of a data collection unit 302 may capture a time ordered series of thermal images of the heating elements of a cooktop 308.

[0189] Reference is next made to FIG. 7, which shows a schematic diagram of an example burner image sequence 702 being assembled into a cooktop event prediction training dataset. Burner image sequence 702 is comprised of a plurality of time-ordered burner thermal images 710. Each burner thermal image 710 may be a thermal image associated with a heating element of a cooktop, which may include the thermal element itself, a cooking implement located on top of a heating element, a food being cooked on the heating element, or any other thermal activity related to that heating element. Burner thermal image 710 may optionally also include other types of image data in addition to thermal data, including RGB image data. The example burner image sequence 702 of FIG. 7 contains 285 total burner thermal images 710 and begins at a first burner thermal image 710-1 and ends at a last burner thermal image 710-285.

[0190] The burner image sequence 702 may contain a series of burner thermal images 710 periodically captured over a period of time. The burner image sequence 702 may be associated with a frame rate. The frame rate may be a constant frame rate. The frame rate may be the frequency at which the images were captured at the time of capture. For example, burner image sequence 702 could have been captured by a thermal imaging sensor saving thermal images at a particular frequency, which may then be considered the frame rate of burner image sequence 702. Consequently, each burner thermal image 710 may provide information relating to the heating element at a different point in time.

[0191] In some embodiments, the plurality of burner thermal images may be extracted from a video of a cooktop. For example, burner thermal images 710 may have been originally recorded in a video in a video encoding format and then converted to animage sequence by extracting still frames from the video. In such situations, the frame rate of burner image sequence 702 may be the frame rate of the video, or it may be lower than the frame rate of the video if a subset of the frames from the video are used.

[0192] Each burner thermal image 710 may depict a condition relating to the heating element at a particular point in time. In some embodiments, at least a portion of the burner thermal images are associated with labels corresponding to the state of the condition of the burner depicted in the frame. For example, as shown on example burner image sequence 702, each burner thermal image 710 may be associated with a cooktop event label 712. A cooktop event label 712 may be a data point indicating the presence of a condition relating to the heating element at a certain point in time. As each burner thermal image 710 is also associated with a point in time, each burner thermal image 710 may be associated with a corresponding cooktop event label 712 through this common temporal relation. For example, a heating element may be on fire at a point in time. The burner thermal image 710 corresponding to that point in time may show high intensity pixel values corresponding to the thermal output of a fire. A cooktop event label 712 corresponding to that point in time may similarly indicate fire, for example, through a data structure wherein a binary 1 is associated with a “fire” condition and a binary 0 associates with a “no fire” condition. The burner thermal image depicting the fire at the point in time may then be associated with the cooktop event label indicating fire at the point in time.

[0193] In some embodiments, the association between the burner thermal images 710 and the cooktop event label 712 may be explicit. For example, an association between each burner thermal image 710 and a cooktop event label may be stored in a data structure. For example, tables or databases that can index each burner thermal image to its corresponding cooktop event label may be used. Alternatively, the cooktop event label may be associated with to the burner thermal image itself through appending the cooktop event label data to the metadata of the image file.

[0194] In some embodiments, the association may also be purely implicit through a time relationship. For example, considering the case of a fire, a fire presence data structure may contain time-series data relating to the presence of fire on a cooktop (e.g., yes or no at each time in the time-series). By temporally aligning the fire presence data structure to the burner image sequence, a burner thermal image may be associated with a data point within the fire presence data structure by selecting a time. As such, a cooktop event label corresponding to the presence of fire may be associated with a burner thermalimage using data from the fire presence data structure. Other further methods of associating the data may be available. For example, data may be available describing a particular interval of time within which fire is present, and is otherwise not present. By examining a particular time and comparing the time to the interval, a determination may be made relating to whether fire is present for any particular burner thermal image, and thus, an implicit cooktop event label relating to fire forthat burner thermal image.

[0195] Referring back to FIG. 5, the method proceeds at 504 with identifying a plurality of timepoints in the burner image sequence. A timepoint may be any indicator of a position within a burner image sequence. In one embodiment, the timepoint may be a time relative to a selected point in the burner image sequence, such as 0.04 seconds, or 2 minutes and 32.54 seconds. As the burner image sequence is associated with a frame rate, each burner thermal image in the burner image sequence can be associated with a time indicating when the burner thermal image appears within the burner image sequence temporally. As such, each timepoint may correspond to a particular burner thermal image. In other embodiments, a timepoint may simply be a frame number that directly references a particular image within a burner image sequence (e.g., frame 1 , 2, 200, 400, etc).

[0196] In the example of FIG. 7, timepoints to, t1 , and t2 are shown. Timepoints to, t1 and t2 align and correspond with particular frames in the burner image sequence 702, particularly, the 40thframe (710-40), 41stframe (710-41), and 42ndframe (710-42) in burner image sequence 702. In the example where burner image sequence begins at 0s at t(-3) and has a frame rate of 8hz / 8fps, first frame 710-1 may correspond to a timepoint of 0.125s (t(-2)), frame 710-40 to 5s, frame 710-41 to 5.125s, frame 710-42 to 5.25s, and frame 710-285, being the 285thframe in the sequence, to 35.625s.

[0197] It should be noted that in the example timing diagram 700 of FIG. 7, the corresponding frame is the frame preceding the timepoint. For example, if the first frame occupies a period of [0, 0.125] and the second frame from [0.125, 0.25], then the timepoint of 0.125 corresponds to the first frame rather than the second frame. However, in some embodiments, the frame taken can be the frame immediately following the timepoint (the second frame) as long as the convention is consistent throughout the process.

[0198] Referring back to FIG. 5, the method proceeds at 506 with, for each timepoint in the burner image sequence, assembling a timepoint training instance, the timepoint training instance comprising: a burner thermal image set corresponding to a window time period corresponding to the timepoint; and a cooktop event labelcorresponding to a state of a condition of the burner at a horizon time corresponding to the timepoint. A timepoint training instance may be a single unit of training data that includes the inputs to a machine learning model and the desired output based on the inputs. Each timepoint training instance is associated with a selected timepoint. A window time period may be any time period over which the data falling on that interval may be provided as input data to a machine learning model to produce an inference, and can be expressed as a time period or a number of frames. A window time period may be a period immediately preceding the timepoint, but may also be any other period defined in relation to the timepoint. A cooktop event label may be any label that indicates the presence of a condition relating to a heating element. A horizon time may be any period of time selected in relation to a selected timepoint which it is desirable to produce an inference relating to cooktop operation at that time, based on the cooktop data observed over the window time period.

[0199] Referring back to the example of FIG. 7, timepoint training instances 704a, 704b, and 704c are shown, corresponding to the selected timepoints at to, t1, and t2. Each timepoint training instance 704 contains burner thermal image set 706, which consist of burner thermal images 710 selected over a window time period 720. In the example shown in FIG. 7, sets 706a, 706b, 706c immediately precede timepoints to, t1 , t2. For example, for timepoint to corresponding to 5 seconds, set 706a contains frames from 0s to 5s, which contains 40 frames at 8 fps, corresponding to frames 710-1 to 710-40. Similarly, timepoint 51 corresponding to 5.125 seconds, set 706b contains frames from 0.125 to 5.125 seconds, corresponding to frames 710-2 to 710-41.

[0200] Timepoint training instances 704 may contain cooktop event labels 712 that each correspond to the state of the cooktop at a horizon time 722 after from the selected timepoint. Horizon time 722 may be defined as a time period or in terms of frames. For example, timepoint training instance 704a contains cooktop event label 712-71, corresponding to the state of the cooktop at a time contemporaneous with frame 710-71 , which is a horizon time 722 later. As shown, cooktop event label 712-71 contains a 0, indicating no fire present. Timepoint training instance 704b contains cooktop event label 712-72, which similarly contains “0”, indicating no fire. However, timepoint training instance 704c, corresponding to timepoint t2, contains cooktop event label 712-73, which contains a “1”. Cooktop event label 712-73 thus indicates the presence of fire at a time t5 that is a horizon time after t2.

[0201] In some embodiments, burner thermal image sets 704 may not immediately precede their respective timepoints. As the window time period 720 can be defined relative to the timepoint in any way, including at a time in the past or in the future relative to the timepoint, burner thermal image sets 704 may be selected from any interval. For example, the interval may begin at 30 frames before the timepoint and end 10 frames after the timepoint. Similarly, the horizon time 722 may be defined in any way relative to the timepoint as long as the corresponding cooktop event label corresponds to a time taking place after the last image in the corresponding burner thermal image set. In the example of FIG. 7, window time period 720 may be a window spanning 5s or 40 frames immediately preceding a selected timepoint, assuming a frame rate of 8hz / 8fps. However, window time period 720 could be any other amount of time or number of frames, depending on the desired input for inference. Similarly, in this example, horizon time 722 is selected to be 3.875s, or 31 frames away, indicating that the desired inference time is 3.875 seconds into the future from the timepoint.

[0202] Each timepoint training instance 704 may be a unit of training data in a machine learning training dataset. Although only three timepoint training instances are shown in diagram 700, many timepoint training instances can be produced from a burner image sequence. For example, if the window time period 720 is defined as the 40 frames immediately preceding the timepoint, and the horizon time 722 is defined as 30 frames following the timepoint, and there are a total of 285 thermal burner images in the burner image sequence 702, then 215 timepoint training instances can be produced from burner image sequence 702. A typical machine learning dataset using in method 500 may contain many thousands or millions timepoint training instances.

[0203] In some embodiments, the burner image sequence may be prepared from a cooktop image sequence. Reference is next made to FIG. 8, which shows a flowchart of a method 800 of preparing a burner image sequence from a cooktop image sequence. The method begins at 802 with imaging a whole cooktop, thereby generating a cooktop image sequence. For example, the setup 300 of FIG. 3 may be employed to generate a cooktop image sequence of a cooktop 308. The data collection unit 302 may be configured in an overhead position to capture thermal images of the entire cooktop 308, generating a cooktop image sequence. The method proceeds at 804 with segmenting the cooktop image sequence into one or more burner image sequences. The images of the cooktop image sequence may show activity occurring on multiple heating elements.These images may be segmented to isolate the individual heating elements and to create sets of burner image sequences, which may depict data corresponding to one heating element. The method proceeds at 806 with selecting the burner image sequence from the one or more burner image sequences. The burner image sequence may then be used in method 500 to assemble a portion of a machine learning training dataset.

[0204] In some embodiments, the burner image sequence is prepared by performing at least one rotation. The rotation may be a rotation of 90, 180, or 270 degrees, but can be a rotation of any degree. The resulting image size may be made consistent with the other images in the data set, which may be done, for example, through operations such as padding and cropping. The rotation of burner image sequences may facilitate greater data variety for the purposes of training. T aking the example of setup 300 of FIG.3, the data collection unit 302 may be fixed above the cooktop 308 in a mounting orientation. As such, any given burner image sequence may capture data relating to a heating element in a single orientation, i.e. , from the point of view of the data collection unit 302. However, thermal image data collected relating to a heating element may be useable in any number of orientations. As such, a single burner image sequence captured in one orientation may produce multiple burner image sequences through rotations about various angles, resulting in a greater variety of training data.

[0205] In some embodiments, the burner image sequence may be prepared by performing incorporating one or more of the following transformations on the burner image sequence: a normalization, a flip, and a de-skew. A normalization may be an adjustment or scaling of the pixel values of the burner thermal images of the burner image sequence. The normalization may be performed to scale pixels values to fall within a range, to center pixel values around a particular value, to increase contrast in the image, or generally to improve performance of the image for use in machine learning. Any number of image normalization techniques may be used, including min-max scaling, selecting a minima or maxima (which may be constrained to a desired or target cooking temperature range or limit), Z-score normalization, histogram equalization, contrast stretching, local contrast normalization, and any other technique that may facilitate improved training performance for machine learning.

[0206] In some embodiments, the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value. For example, a reference pixel value can be determined based on an average value of the image, theimage sequence, or the entire dataset. Based on this pixel value, the pixel values of each burner thermal image may be adjusted relative to that value, for example, based on ensuring the values fall within one standard deviation of the value. In some embodiments, the average temperature is based on the cooktop image sequence dataset.

[0207] A flip may be a linear transformation of a burner thermal image that involves a reflection across a line. The line may be, for example, across lines horizontally or vertically bisecting the image. However, the line can be defined as any line.

[0208] A de-skew may be a transformation of a burner thermal image that involves a perspective correction. For example, a square cooktop may be imaged as a trapezoid due to the mounting angle of the thermal sensor relative to the surface of the cooktop, for instance, as shown in FIG. 6. A de-skew may be performed on the image to correct the perspective of the captured image into a square.

[0209] It should be noted that the frame rate of burner image sequence 702 may not necessarily be the same as a source frame rate, i.e., the frequency at which the images are captured or the frame rate of a source video. The frame rate may be selected at a value that differs from the source frame rate, as long as the frame rate remains sufficiently responsive to the rate of change observed with respect to typical cooktop activities such as cooking. The frame rate may be varied for various considerations, including computation resources, model size, training data size, hardware limitations, performance, efficiency, and any other such similar considerations. For example, burner image sequence 702 may be derived from a video originally recorded at 24 fps. However, one in every three frames of the video may be used to reduce labelling requirements. Thus, the frame rate of image sequence 702 may be 8fps instead of 24. The inventors have found that, for some embodiments, 8fps is a sufficient frame rate to achieve satisfactory training and inference performance, but a greater or lesser frame rate may be used.

[0210] Similarly, the timepoints selected may not necessarily be selected at a frequency equal to the frame rate. In some embodiments, the timepoints are selected at a timepoint frequency, the timepoint frequency being equal to the frame rate. For example, if the frame rate of a burner image sequence is 8 fps, then 8 timepoints in the burner image sequence may be selected for each second depicted in the burner image sequence. In some embodiments, the timepoint frequency may be less than the frame rate. For example, if the frame rate of a burner image sequence is 8 fps, then the timepointfrequency may be selected as 4fps. For example, a burner image sequence may contain 200 burner thermal images, with a window of 40 images and a horizon of 30 images. If the cooktop event labels are all associated with an image in the burner image sequence, a maximum of 130 timepoints can be selected that can be assembled into timepoint training instances. If the timepoint frequency is equal to the framerate, 130 timepoints can be selected and 130 timepoint training instances can be assembled. However, if only every other available timepoint (making the timepoint frequence half of the frame rate) is selected in the interest of, for example, expediency of processing training data, 65 timepoints may be selected instead, and 65 timepoint training instances may be assembled.

[0211] In some embodiments, the conditions comprise at least one of: fire, boil-over, operating status, and pot-too-hot, and uncovered burner. The boil-over condition may indicate the presence of a liquid over-flowing or spilling out of a cooking implement on the heating element. The pot-too-hot condition may indicate excessive temperature of a cooking implement on top of a heating element, such as a pan that has been left to preheat for too long. Operating status may, for example, indicate whether or not a heating element is in use. Operating status may include other aspects of the operation or operating conditions of a heating element, such as the type of cooking being undertaken, the setting of the burner, the type of individual operating (or apparently operating) the burner (for example, an adult or a child). Uncovered burner may indicate a heating element left on without any cooking implement or food on top of the burner, which may be potentially indicative of a user forgetting to turn the heating element off.

[0212] A cooktop event label may be obtained from various sources. In some embodiments, at least a portion of the cooktop event labels is received from a human performing manual labelling on the cooktop image sequence. For example, in the example burner image sequence 702 of FIG. 7, each burner thermal image 750 may be associated with a cooktop event label 760. A human may have reviewed at least a portion of the burner thermal images 750 and manually determined, for example, the presence of fire within the burner image through human judgement. For example, the human may review the thermal images directly. As another example, the human may be reviewing an RGB image sequence or video taken contemporaneously with the thermal image sequence and be generating the label based thereon.

[0213] In some embodiments, at least a portion of the cooktop event labels is received from a computing system performing automated labelling of the cooktop image sequence. Computer vision techniques may be used. For example, a machine learning model configured to label data using RGB image data can review RGB image or video data corresponding to a point in time and produce a cooktop event label indicating the presence of fire for that point in time. However, other automated techniques can also be used to similar effect. For example, an apparatus containing a heat detector and smoke detector can be used to automatically detect the occurrence of a fire at a point in time and produce a corresponding cooktop event label forthat point in time automatically.

[0214] In some embodiments, a combination of manual labelling and automated labelling may be used. For example, the automated labelling system may produce labels for the collected data, but flag certain labels if a degree of certainty drops below a certain threshold. Then, a human can manually intervene and review the labels for accuracy. Alternatively, an automated system may produce labels for the collected data, but a human may manually review all labels indicating the presence of fire to ensure accuracy for only a selected category of labels.

[0215] In some embodiments, the cooktop image sequence is an element within a cooktop image sequence dataset, the cooktop image dataset comprising a plurality of elements. For example, the cooktop image sequence dataset may be a dataset that contains thousands to millions of unique cooktop image sequences.

[0216] The cooktop event prediction training dataset can be used to train one or more machine learning models for predicting a cooktop event. Reference is next made to FIG. 9, which shows a flowchart of a method 900 for generating a cooktop event prediction model for predicting a cooktop event.

[0217] Method 900 may be performed on a computing device capable of executing algorithms for training machine learning models. Any computing device containing sufficient processing capability for generating training machine learning models can be used. An example computing device 1200 that can be used to carry out method 900 is shown in FIG. 12. Computing device 1200 can include a processor unit 1208, power unit 1202, memory 1210, I / O unit 1214, and a communications unit 1204. Power unit 1202 may be configured to provide power to the computing device and hardware components of the device 1200. I / O unit can include hardware for accepting inputs and generating outputs, which may be configured to interface with input means such as mice, keyboards,touchscreens, and output means such as monitors, displays, speakers, and more. Memory 1210 can provide storage capabilities for data and programs used by computing device 1200. For example, memory 1210 can include flash memory, hard drives, solid state drives, RAM, removable memory, and more. Processor unit 1208 can include any processor capable of executing machine instructions for training a machine learning model. Processor unit 1208 can include a central processing unit and a graphics processing unit.

[0218] Memory unit 1210 can have stored upon it one or more machine learning models 1220. The machine learning models can be any model that can be trained or finetuned and operated to produce inferences based on the training data. For example, the machine learning model may include one or more components such as neural networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory neural networks (LSTMs), transformers, decision trees, random forests, support vector machines, autoencoders, and more. It should be appreciated that any model architecture capable of accepting as input a series of images or frames, and producing a classification as output can be used. In one non-limiting example, CNN-based layers can be used to embed the input, with transformer-based bodies for processing the input, fed into a classification head to produce a classification output.

[0219] The machine learning models 1220 may be initially in an untrained state, i.e., the weights may be initialized at an initial value or a random value. The machine learning models 1220 can also be pre-trained and be capable of being fine-tuned based on data. Processor unit 1208 can be operated to execute algorithms for training the machine learning models 1220 on training data.

[0220] The method begins, at 902, with providing, at a memory, a cooktop event prediction training dataset. The cooktop event prediction training dataset can be contain multiple instances of training data, containing examples of the input data and the expected outcome. Each instance of training data may be referred to as a timepoint training instance. Each timepoint training instance may contain a set of burner thermal images. The burner thermal images may depict individual burners of a cooktop over a period of time. For example, the burner thermal images may show burners from cooktop 308 of FIG. 3. The time period over which the set of burner thermal images depicts the burners may be a window time period. For example, the window time period could be a time period of between 2-60 seconds, such as 5 seconds, 10 seconds, 24 seconds, or 30 seconds.Alternatively, the window time period could be defined in terms of a number of images or frames in a sequence, which may in turn correspond to a time based on the framerate of the sequence. The timepoint training instances may, as an example, be the timepoint training instances 704 assembled using method 500 (FIG. 5) and as shown in FIG. 7. However, it will be appreciated that the cooktop event prediction training dataset can be obtained from any source and assembled in any manner, as long as they contain data instances comprising sets of burner thermal images corresponding to cooktop event labels.

[0221] Each timepoint training instance may include a training cooktop event label corresponding to a state of a condition of the depicted burner at a horizon time after the window time period. The horizon time may be any time or any number of image / frames following the time depicted in the set burner thermal image. For example, the horizon time period may be a time period between 0 to 120 seconds, or longer, such as 10 seconds, 20 seconds, 24 seconds, 30 seconds or 60 seconds. In such a case, the training cooktop event label may correspond to the state of a condition of the burner 10 seconds after the state depicted in the burner thermal images. The conditions can include at least one of: fire, boil-over, operating status, and pot-too-hot, and uncovered burner, and the state of the condition may be a binary T or ‘0’ corresponding to whether the condition is present.

[0222] The method proceeds, at 904, with operating, at a processor, a machine learning model to produce one or more cooktop event labels based on one or more sets of training burner thermal images corresponding to one or more timepoint training instances. The machine learning model can be any machine learning model configured to take in sets of training burner thermal images as input and produce classification outputs corresponding to a state of a condition. The machine learning models may be capable of being fitted based on the timepoint training instances. For example, the machine learning models can be models 1220 stored on computing device 1200 of FIG.12.

[0223] The machine learning model may be used to make inferences based on one or more sets of training burner thermal images. The machine learning models can be untrained models or pre-trained models. The machine learning models may accept a set of training burner thermal image as input and produce a classification output corresponding to the state of the depicted burner at a horizon time after the time depicted in the set of burner thermal images.

[0224] The method proceeds, at 906, with modifying, at the processor, one or more parameters of the machine learning model based on a comparison between the one or more inferred cooktop event labels and one or more training cooktop event labels corresponding to the one or more timepoint training instances. The machine learning model can contain one or more parameters that can be modified to improve inference accuracy. For example, for neural network-based models, the parameters can include model weights. As another example, for tree-based methods, the parameters can include tree nodes. The modification of the parameters can be done in accordance with one or more optimization techniques. The optimization techniques may optimize the inference accuracy, or reduce an inference error of the model. The optimization techniques may be selected based on the model architecture. For example, gradient-based optimization such as stochastic gradient descent, Adam, AdaGrad, can be used. Alternatively, for treebased models, tree-based methods such as XGBoost, LightGBM, CART (Classification and Regression Trees) can be used.

[0225] In some embodiments, the method further includes dividing, at the processor, a portion of the cooktop event prediction training dataset into a model training dataset, model validation dataset, and model testing dataset. The modifying the one or more parameters of the machine learning model can further include modifying, at the processor, one or more model weights of the machine learning model based on the model training dataset, and modifying, at the processor, one or more hyperparameters associated with the machine learning model based on the model validation dataset. For example, the cooktop event prediction training dataset can be divided into thirds, or in a 50%:30%:20% distribution, into the model training dataset, model validation dataset, and model testing dataset. The model training dataset can be used to adjust the model weights based on inference performance using training methods as described above. The model validation dataset can be used to adjust hyperparameters based on the, such as learning rate, batch size, activation functions, and any other hyperparameter associated with the model or with the training method of the model, in order to further tune the model.

[0226] In some embodiments, the method further comprises testing, at the processor, the machine learning model based on the testing dataset once a training error has reached a threshold. The testing dataset may be any portion of the cooktop event prediction dataset that is not used in training or tuning the model and is reserved for usein testing the model after training is completed. The testing dataset may be used to produce an independent evaluation of the accuracy of inference after training.

[0227] In some embodiments, the set of training burner thermal images are extracted from a cooktop video. For example, the set of training burner thermal images may be image stills taken from a video of a cooktop, such as described with reference to the thermal burner images 710 of FIG. 7.

[0228] In some embodiments, the training burner image sequence is preprocessed by performing at least one of the following transformations on the training burner image sequence: a normalization, a flip, and a de-skew. In some embodiments, the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value. In some embodiments, the training burner image sequence is prepared by performing at least one rotation.

[0229] Reference is next made to FIG. 10, which shows a flowchart of a method 1000 for predicting a cooktop event at a cooktop. Method 1000 can be performed by any system capable of collecting data about the cooktop with a sensor and operating a machine learning model to perform inference using the collected data. For example, method 1000 can be performed by cooktop monitoring unit 102 on cooktop 104 (see FIG.1). Cooktop monitoring unit may use sensors 214, controller 208, and prediction models 220 to perform method 900 to predict a cooktop event at cooktop 104. As described with reference to FIG. 1 , a cooktop event can be any activity or situation on a cooktop that is or may result in an unsafe use of the cooktop or in an unsafe cooking condition. For example, cooktop events may include, but are not limited to, fire, unattended cooking, high temperature uncovered heating element(s), high temperature cookware, high temperature food contained in cookware, burning food, undercooked food, heating element left on and child operating cooktop.

[0230] The method begins, at 1002, with collecting, at a sensor, a burner image sequence comprising a plurality of burner thermal images. The plurality of burner thermal images may depict a burner on a cooktop, such as a heating element 114 of cooktop 104. The sensor can be sensors 214 of cooktop monitoring unit 102. The burner image sequence can be a sequence of images or images extracted from frames of a video. For example, sensors 214 can continuously image cooktop 104, and stream the images continuously to controller 208. The burner image sequence can be regular images of the burner, or can be thermal images. For example, sensors 214 may be thermal imagingsensors configured to generate thermal images. In some embodiments, a window of a number of images is maintained. For example, the most recent 40 images (or, some number of seconds of video data) received from sensors 214 can be kept in memory and used as the burner image sequence.

[0231] The method proceeds, at 1004, with generating, at a controller, a cooktop event classification based on the plurality of burner thermal images using a cooktop event prediction model, the cooktop event classification corresponding to a predicted state of a condition of the cooktop corresponding to a horizon time after the burner image sequence. The controller can be a controller capable of operating a trained machine learning model to produce an inference. For example, controller 208 of FIG. 2 can be used. The cooktop event prediction model may be a trained machine learning model configured to take, as input, the plurality of burner thermal images and produce, as output, a cooktop event classification. For example, prediction models 220, stored on memory 210 of device 102 can be used.

[0232] The classification can correspond to the predicted state of a condition of the cooktop. The conditions of the cooktop could include, for example, one or more of a fire, unattended cooking, high temperature uncovered heating element(s), high temperature cookware, high temperature food contained in cookware, burning food, undercooked food, heating element left on and child operating cooktop. The state of the condition can be whether or not the condition is predicted to be present. For example, the classification output can take the form of a binary ‘T or ‘0’ corresponding to whether a fire is predicted to occur based on the input burner image sequence.

[0233] The cooktop event classification can correspond to a horizon time after the burner image sequence. The horizon time can be a time (e.g. a number of seconds) after the burner image sequence. For example, the classification can correspond to a predicted state of a condition 10 seconds, 20 seconds, or 30 seconds after the state of the burner depicted in the burner image sequence, depending on how the model is configured.

[0234] The model may be generated using training data generated using example burner image sequences. For example, the model can be a model generated using method 900, using a training dataset assembled using method 500.

[0235] In some embodiments, further processing may be performed on the captured images from the sensor before generating the classification. For example, insome embodiments, the burner image sequence can be prepared by imaging the cooktop, thereby generating a cooktop image sequence, segmenting the cooktop image sequence into one or more burner image sequences, and selecting the burner image sequence from the one or more burner image sequences. For example, sensor 214 may image the entirety of cooktop 104, which may contain of a number of heating elements 114. However, the models used may be configured to process only burner thermal image sequences containing only one heating element at a time. Thus, each frame of the overall image sequence may be processed by segmenting each frame into individual burner images. A burner image sequence that depicts only one burner can then be used at a time.

[0236] In some embodiments, the further processing can include transformations such as a normalization, a flip, a de-skew, and / or a rotation, which may allow the input images to match the training data more closely. The normalization can involve scaling a pixel value of the plurality of burner thermal images based on an average value.

[0237] FIG. 11 shows a diagram depicting steps 1002 and 1004. A burner image sequence 1102 consisting of plurality of burner thermal images 1104 is provided as input to trained machine learning model 1110. Model 1110 then produces a classification 1120 corresponding to the burner image sequence 1102, which relates to the predicted state of a condition at some time after the state of the burner depicted in the burner image sequence.

[0238] The method proceeds, at 1006, with initiating, at the controller, an action based on the cooktop event classification. The action may be a warning action, a preventative action, an emergency action or another type of action that may avoid, reduce or mitigate the occurrence of a cooktop event or the effect of a cooktop event.

[0239] A warning action can be any action the controller is capable of initiating based on the classification that may provide some warning or remedial action against a predicted state of a condition. For example, the controller may be controller 208 of monitoring device 102, as shown in FIG. 2. Controller 208 may be capable of activating an indicator module 212 to produce a visual or audio notification. For example, a lighting or activated colored LED light can be activated to warn a user if a fire is predicted. Additionally or alternatively, a beeping can be initiated. In some embodiments, the warning action can include activating a fire alarm system. For example, monitoring device 102 may be connected to a fire alarm system through communications module. As such,if a fire is predicted, the fire alarm system can be activated to notify occupants to evacuate, or to trigger one or more remedial measures such as sprinklers and / or other fire suppression systems.

[0240] A damage or injury reduction or mitigation action may include disconnecting power to a cooktop or to a heating element with a cooktop. For this purpose, a power disconnect module may be coupled to the cooktop power supply or to the heating element. The power disconnect module may include a switch that can activated under the control of the controller. The controller may be coupled to the power disconnect module through a wired or wireless connection. The controller may disconnect power to the cooktop or to one or more heating elements based on a cooktop event classification. For example, if the cooktop event classification indicates or predicts an urgent situation such as a fire on the cooktop, the controller may disconnect power to the cooktop or a heating element. The switch in the power disconnect may be operable in response to a power disconnect signal sent by the controller. For example, in some embodiments, the switch may be coupled to the controller through a switch controller that receives a wired or wireless signal controller and in response disconnects power to the cooktop or heating element. The switch controller may be a relay, processor or any other device that can receive a power controls signal from the controller and activate the switch.

[0241] In some embodiments, continued monitoring of a cooktop may indicate that a cooktop event that previously led to power being disconnected from the cooktop or a heating element is likely to occur. Optionally, the controller may be configured to reconnect power to the cooktop or heating element by sending a power reconnect signal to the power disconnect module.

[0242] The action may vary depending on the specific classification output. Different levels of notification and numbers of notification means can be activated depending on the severity of the classification. As an example, if only a “pot-too-hot” classification is detected, then the warning action can simply be a visual notification. However, if a “fire” condition is predicted, then the warning action may additionally include auditory notifications and / or triggering of a fire alarm system and may include a power disconnect action.In some embodiments, the action can include multiple stages. The stages may provide for gentler notification at an initial time and may escalate under certain conditions. The conditions can be any condition in which it may be desired to escalate the warning action,such as lack of action, or a change in status. For example, if a fire is predicted to occur, an initial stage may involve only a visual notification. If a fire continues to be predicted with no intervention detected, then a second stage can involve additional warnings, such as auditory warnings or activation of an external fire alarm system. Additionally, if an initial classification was “pot-too-hot”, a first stage may involve only a visual warning. However, if the classification changes to “fire” at a later point in time, a second stage may be activated that initiates an auditory notification, a power disconnect action or both.

[0243] Referring again to Figure 7, in some embodiments, one or more models may be trained with a data set in which the label is a temperature value. Each timepoint training instance 704 in the training dataset may include a burner thermal image set 706 of training burner thermal images 710 corresponding to a window time period 720 and a label 712 corresponding to a temperature of a training burner at the end of a horizon time period 722. Such a training data set may be used to train a machine learning model to generate a cooktop even prediction model to predict the temperature of or at a burner or heating element. During method 1000, the trained model may be used to predict the temperature at a heating element and actions may be taken in response to the predicted temperature. For example, if the predicted temperature at a heating element exceeds an allowable or safe temperature based on the operating condition or operating status of a cooktop or heating element. For example, if the operating condition or operating status indicates that a heating element is being use to boil water, the safe temperature may be just above the boiling point of water. If the operating condition or operating status indicates that a heating element is being used to try foods in oil, the safe temperature may slightly above the desired oil temperature.While the above description describes features of example embodiments, it will be appreciated that some features and / or functions of the described embodiments are susceptible to modification without departing from the spirit and principles of operation of the described embodiments. For example, the various characteristics which are described by means of the represented embodiments or examples may be selectively combined with each other. Accordingly, what has been described above is intended to be illustrative of the claimed concept and non-limiting. It will be understood by persons skilled in the art that other variants and modifications may be made without departing from the scope of the invention as defined in the claims appended hereto. The scope of the claims shouldnot be limited by the preferred embodiments and examples, but should be given the broadest interpretation consistent with the description as a whole.

Claims

CLAIMS:

1. A method for assembling a cooktop event prediction training dataset, the method comprising:providing a burner image sequence, the burner image sequence comprising a plurality of burner thermal images;identifying, at a processor, a plurality of timepoints in the burner image sequence; andfor each timepoint in the burner image sequence, assembling, at the processor, a timepoint training instance, the timepoint training instance comprising:i) a set of burner thermal images corresponding to a window time period corresponding to the timepoint; andii) a cooktop event label corresponding to a state of a condition of the burner at a horizon time corresponding to the timepoint.

2. The method of claim 1 , wherein the burner image sequence is prepared by:imaging a whole cooktop, thereby generating a cooktop image sequence; segmenting the cooktop image sequence into one or more burner image sequences; andselecting the burner image sequence from the one or more burner image sequences.

3. The method of claim 1 , wherein the burner image sequence is prepared by performing at least one of the following transformations on the burner image sequence: a normalization, a flip, a de-skew.

4. The method of any one of claims 1 to 3, wherein the burner image sequence is prepared by adding noise to the burner image sequence.

5. The method of claim 4 wherein the added noise is one or more types of noise selected from the group consisting of: Gaussian noise, Gaussian blur noise, salt-and-pepper noise, Poisson noise, speckle noise, motion blur noise and color jitter noise.

6. The method of claim 1 , wherein at least a portion of the burner thermal images are labelled corresponding to the state of the condition of the burner depicted in the frame.

7. The method of claim 1 , wherein the burner image sequence is prepared by performing at least one rotation.

8. The method of claim 1 , wherein the plurality of burner thermal images are extracted from a cooktop video.

9. The method of claim 1 , wherein the plurality of burner frames are captured at a constant frame rate.

10. The method of claim 9, wherein the timepoints are selected at a timepoint frequency, the timepoint frequency being equal to the frame rate.

11. The method of claim 9, wherein the timepoints are selected at a timepoint frequency, the timepoint frequency being less than the frame rate.

12. The method of claim 6, wherein the conditions comprise at least one of: fire, boil-over, operating status, pot-too-hot, and uncovered burner.

13. The method of claim 6, wherein the conditions comprise a temperature corresponding to the burner.

14. The method of claim 1, wherein at least a portion of the cooktop event labels is received from a human performing manual labelling on the cooktop image sequence.

15. The method of claim 1, wherein at least a portion of the cooktop event labels is received from a computing system performing automated labelling of the cooktop image sequence using computer-vision-based techniques.

16. The method of claim 1, wherein the cooktop image sequence is an element within a cooktop image sequence dataset, the cooktop image dataset comprising a plurality of elements.

17. The method of claim 3, wherein the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

18. The method of claim 17, wherein the average value is based on the cooktop image sequence dataset.

19. A system for assembling a cooktop event prediction training dataset from a plurality of cooktop prediction data series, the system comprising:a memory;a processor in communication with the memory, the processor configured to:provide a burner image sequence, the burner image sequence comprising a plurality of burner thermal images;identify a plurality of timepoints in the burner image sequence; and for each timepoint in the burner image sequence, assembling a timepoint training instance, the timepoint training instance comprising:a set of burner thermal images corresponding to a window time period corresponding to the timepoint; anda cooktop event label corresponding to a state of a condition of the burner at a horizon time corresponding to the timepoint.

20. The system of claim 19, wherein the burner image sequence is prepared by:imaging a whole cooktop, thereby generating a cooktop image sequence; segmenting the cooktop image sequence into one or more burner image sequences; andselecting the burner image sequence from the one or more burner image sequences.

21. The method of claim 19, wherein the burner image sequence is prepared by performing at least one of the following transformations on the burner image sequence: a normalization, a flip, and a de-skew.

22. The method of any one of claims 19 to 21, wherein the burner image sequence is prepared by adding noise to the burner image sequence.

23. The method of claim 22 wherein the added noise is one or more types of noise selected from the group consisting of: Gaussian noise, Gaussian blur noise, salt-and-pepper noise, Poisson noise, speckle noise, motion blur noise and color jitter noise.

24. The system of claim 19, wherein at least a portion of the burner thermal images are labelled corresponding to the state of the condition of the burner depicted in the frame.

25. The system of claim 19, wherein the burner image sequence is prepared by performing at least one rotation.

26. The system of claim 19, wherein the plurality of burner thermal images are extracted from a cooktop video.

27. The system of claim 19 wherein the plurality of burner frames are captured at a constant frame rate.

28. The system of claim 27, wherein the timepoints are selected at a timepoint frequency, the timepoint frequency being equal to the frame rate.

29. The system of claim 27, wherein the timepoints are selected at a timepoint frequency, the timepoint frequency being less than the frame rate.

30. The system of claim 24, wherein the conditions comprise at least one of: fire, boil-over, operating status, pot-too-hot, and uncovered burner.

31. The method of claim 24, wherein the conditions comprise a temperature corresponding to the burner.

32. The system of claim 19, wherein at least a portion of the cooktop event labels is received from a human performing manual labelling on the cooktop image sequence.

33. The system of claim 19, wherein at least a portion of the cooktop event labels is received from a computing system performing automated labelling of the cooktop image sequence using computer-vision-based techniques.

34. The system of claim 19, wherein the cooktop image sequence is an element within a cooktop image sequence dataset, the cooktop image dataset comprising a plurality of elements.

35. The system of claim 21 , wherein the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

36. The system of claim 35, wherein the average value is based on the cooktop image sequence dataset.

37. A method for predicting a cooktop event at a cooktop, the method comprising:collecting, at a sensor, a burner image sequence comprising a plurality of burner thermal images;generating, at a controller, a cooktop event classification based on the plurality of burner thermal images using a cooktop event prediction model, the cooktop event classification corresponding to a predicted state of a condition of the cooktop corresponding to a horizon time after the burner image sequence;initiating, at the controller, a warning action based on the cooktop event classification.

38. The method of claim 37, wherein the warning action comprises one or more of: an auditory notification and a visual notification.

39. The method of claim 37, wherein the warning action comprises activating a fire alarm system.

40. The method of claim 37, wherein the burner image sequence is prepared by:imaging the cooktop, thereby generating a cooktop image sequence; segmenting the cooktop image sequence into one or more burner image sequences; andselecting the burner image sequence from the one or more burner image sequences.

41. The method of claim 37, wherein the burner image sequence is prepared by performing at least one of the following transformations on the burner image sequence: a normalization, a flip, and a de-skew.

42. The method of any one of claims 37 to 41, wherein the burner image sequence is prepared by adding noise to the burner image sequence.

43. The method of claim 42 wherein the added noise is one or more types of noise selected from the group consisting of: Gaussian noise, Gaussian blur noise, salt-and-pepper noise, Poisson noise, speckle noise, motion blur noise and color jitter noise.

44. The method of claim 41 , wherein the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

45. The method of claim 37, wherein the burner image sequence is prepared by performing at least one rotation.

46. The method of claim 37, wherein the plurality of burner thermal images are extracted from a cooktop video.

47. The method of claim 37, wherein the sensor comprises a thermal imaging sensor.

48. The method of claim 37, wherein the conditions comprise at least one of: fire, boil-over, operating status, pot-too-hot, and uncovered burner.

49. The method of claim 37, wherein the conditions comprise a temperature corresponding to the burner.

50. A system for predicting a cooktop event at a cooktop, the system comprising:a sensor, configured to collect a burner image sequence comprising a plurality of burner thermal images;a memory unit, having stored thereon:i) a cooktop event prediction model;a controller, configured to:i) generate, using the cooktop event prediction model, a cooktop event label based on the plurality of burner thermal images, wherein the cooktop event label corresponds to a predicted state of a condition of the cooktop corresponding to a horizon time after the burner image sequence; andii) initiate a warning action based on the cooktop event label.

51. The system of claim 50, wherein the warning action comprises one or more of: an auditory notification and a visual notification.

52. The system of claim 50, wherein the warning action comprises activating a fire alarm system.

53. The system of claim 50, wherein the burner image sequence is prepared by:imaging the cooktop, thereby generating a cooktop image sequence; segmenting the cooktop image sequence into one or more burner image sequences; andselecting the burner image sequence from the one or more burner image sequences.

54. The system of claim 50, wherein the burner image sequence is prepared by performing at least one of the following transformations on the burner image sequence: a normalization, a flip, and a de-skew.

55. The system of claim 54, wherein the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

56. The system of claim 50, wherein the burner image sequence is prepared by performing at least one rotation.

57. The system of claim 50, wherein the plurality of burner thermal images are extracted from a cooktop video.

58. The system of claim 50, wherein the sensor comprises a thermal imaging sensor.

59. The system of claim 50, wherein the conditions comprise at least one of: fire, boil-over, operating status, pot-too-hot, and uncovered burner.

60. The system of claim 50, wherein the conditions comprise a temperature corresponding to the burner.

61. A method for generating a cooktop event prediction model for predicting a cooktop event, comprising:providing, at a memory, a cooktop event prediction training dataset, the cooktop event prediction training dataset comprising a plurality of timepoint training instances, each timepoint training instance comprising:i) a set of training burner thermal images corresponding to a window time period; andii) a training cooktop event label corresponding to a state of a condition of a training burner at a horizon time after the window time period;operating, at the processor, a machine learning model to produce one or more cooktop event labels based on one or more sets of training burner thermal images corresponding to one or more timepoint training instances; andmodifying, at the processor, one or more parameters of the machine learning model based on a comparison between the one or more inferred cooktop event labels and one or more training cooktop event labels corresponding to the one or more timepoint training instances.

62. The method of claim 61 , further comprising:dividing, at the processor, a portion of the cooktop event prediction training dataset into a model training dataset, model validation dataset, and model testing dataset.

63. The method of claim 62, wherein the modifying the one or more parameters of the machine learning model further comprises:modifying, at the processor, one or more model weights of the machine learning model based on the model training dataset; andmodifying, at the processor, one or more hyperparameters associated with machine learning model based on the model validation dataset.

64. The method of claim 61, further comprising testing, at the processor, the machine learning model based on the testing dataset once a training error has reached a threshold.

65. The method of claim 61, wherein the set of training burner thermal images are extracted from a cooktop video.

66. The method of claim 61 , wherein the training burner image sequence is preprocessed by performing at least one of the following transformations on the training burner image sequence: a normalization, a flip, and a de-skew.

67. The method of claim 66, wherein the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

68. The method of claim 61 , wherein the training burner image sequence is prepared by performing at least one rotation.

69. A system for generating a cooktop event prediction model for predicting a cooktop event, comprising:a memory, having stored thereon a cooktop event prediction training dataset, the cooktop event prediction training dataset comprising a plurality of timepoint training instances, each timepoint training instance comprising:i) a set of training burner thermal images corresponding to a window time period; andii) a training cooktop event label corresponding to a state of a condition of a training burner at a horizon time after the window time period;a processor, configured to:i) produce, using a machine learning model, one or more cooktop event labels based on one or more sets of training burner thermal images corresponding to one or more timepoint training instances; andii) modify one or more parameters of the machine learning model based on a comparison between the one or more inferred cooktop event labels and one or more training cooktop event labels corresponding to the one or more timepoint training instances.

70. The system of claim 69, wherein the processor is further configured to:divide a portion of the cooktop event prediction training dataset into a model training dataset, model validation dataset, and model testing dataset.

71. The system of claim 70, wherein the modifying the one or more parameters of the machine learning model further comprises:modifying, at the processor, one or more model weights of the machine learning model based on the model training dataset; andmodifying, at the processor, one or more hyperparameters associated with machine learning model based on the model validation dataset.

72. The system of claim 69, wherein the processor is further configured to test, at the processor, the machine learning model based on the testing dataset once a training error has reached a threshold.

73. The system of claim 69, wherein the set of training burner thermal images are extracted from a cooktop video.

74. The system of claim 69, wherein the training burner image sequence is preprocessed by performing at least one of the following transformations on the training burner image sequence: a normalization, a flip, and a de-skew.

75. The system of claim 74, wherein the normalization comprises scaling a pixel value of the plurality of burner thermal images based on an average value.

76. The system of claim 69, wherein the training burner image sequence is prepared by performing at least one rotation.

77. A method for assembling a cooktop-event prediction training dataset, the method comprising:providing a burner image sequence comprising a plurality of time-ordered burner images derived from sensor data;identifying, by a processor, a plurality of timepoints within the burner image sequence; andfor each timepoint, assembling a timepoint training instance comprising:(i) an input window including a sequence of burner images associated with a window time period defined relative to the timepoint; and(ii) a cooktop-event label corresponding to a state of a condition at a horizon time defined relative to the timepoint.

78. The method of claim 77, wherein the burner image sequence is generated by: imaging an entire cooktop to obtain a cooktop image sequence; segmenting the cooktop image sequence into one or more burner-specific image sequences; and selecting one of the burner-specific image sequences as the burner image sequence.

79. The method of claim 76, wherein the burner images comprise thermal images, visible-light images, infrared images, or a combination thereof obtained from one or more sensors.

80. The method of claim 76, further comprising performing one or more transformations on the burner image sequence selected from: normalization, flipping, rotation, and de-skewing.

81. The method of claim 80, wherein the normalization comprises scaling pixel values based on an average value, the average value computed per image, per sequence, or over a dataset of sequences.

82. The method of claim 80, wherein the rotation is 90°, 180°, or 270°.

83. The method of claim 76, further comprising adding noise to the burner image sequence, the noise comprising one or more of: Gaussian noise, Gaussian blur, salt-and-pepper, Poisson noise, speckle noise, motion blur, and color jitter.

84. The method of claim 76, wherein at least a portion of the cooktop-event labels are generated by manual labeling, automated labeling using computer-vision techniques, or a combination thereof.

85. The method of claim 76, wherein the burner images are frames extracted from a video captured at a source frame rate, and wherein the burner image sequence is sub-sampled from the source frame rate.

86. The method of claim 76, wherein the timepoints are selected at a timepoint frequency equal to the burner image sequence frame rate.

87. The method of claim 76, wherein the timepoints are selected at a timepoint frequency less than the burner image sequence frame rate.

88. The method of claim 76, wherein the condition comprises at least one of: fire, boil-over, operating status, pot-too-hot, uncovered burner, or a temperature value.

89. The method of claim 76, wherein the input window comprises a sliding window of the most recent N frames stored in a buffer.

90. The method of claim 78, wherein segmenting comprises perspective correction or homography-based de-skewing prior to isolating burner regions.

91. The method of claim 76, wherein the cooktop-event label is binary, multi-class, multi-label, or a regression value.

92. The method of claim 76, further comprising: creating timepoint training instances that use images for a plurality of burners concurrently, the input window including frames from two or more burner regions.

93. The method of claim 76, wherein the horizon time is constrained to occur after a last frame of the input window.

94. The method of claim 76, further comprising storing the assembled timepoint training instances in a training dataset file or database structure indexed by sequence identifier, timepoint, window start, window end, and horizon.

95. The method of claim 76, wherein the sensor data further comprises at least one of: depth data, humidity, ambient temperature, utility draw, audio, or electromagnetic emissions, and the input window includes corresponding time-aligned features.

96. The method of claim 76, wherein normalization parameters are computed at the dataset level and applied consistently across all sequences.

97. A method for generating a cooktop-event prediction model, the method comprising:• providing a training dataset comprising a plurality of timepoint training instances, each instance including an input window of burner images and a corresponding cooktop-event label at a horizon time;• operating a machine learning model to produce one or more predicted labels from the input windows; and• modifying one or more model parameters based on a comparison between the predicted labels and the corresponding cooktop-event labels.

98. The method of claim 97, further comprising partitioning the training dataset into a model-training subset, a validation subset, and a testing subset.

99. The method of claim 97, wherein modifying the model parameters comprises updating model weights using a gradient-based optimizer.

100. The method of claim 98, further comprising tuning one or more hyperparameters based on performance on the validation subset.

101. The method of claim 98, further comprising evaluating the trained model on the testing subset after a training error satisfies a stopping criterion.

102. The method of claim 97, wherein the machine learning model comprises at least one of: a convolutional neural network, a recurrent neural network, a long short-term memory network, a transformer, a 3D-CNN, a support vector machine, a decision tree, a random forest, or an autoencoder.

103. The method of claim 97, wherein the input window is encoded into per-frame feature embeddings which are aggregated temporally by a sequence model.

104. The method of claim 97, wherein the cooktop-event label comprises a temperature value predicted at the horizon time, and the modifying comprises minimizing a regression loss.

105. The method of claim 97, wherein the input window includes images from two or more burners and the machine learning model learns cross-burner correlations.

106. The method of claim 97, wherein at least a portion of the training dataset is augmented using transformations comprising normalization, flipping, rotation, de-skewing, and noise injection.

107. A method for predicting a cooktop event at a cooktop, the method comprising:• collecting, by at least one sensor, a burner image sequence comprising a plurality of time-ordered burner images;• maintaining, by a controller, an input window comprising a most-recent sequence of burner images;• generating, by the controller using a trained cooktop-event prediction model, a cooktop-event classification or regression output corresponding to a predicted state of a condition at a horizon time; and• initiating, by the controller, an action based on the generated output.

108. The method of claim 107, wherein the action comprises issuing a visual notification, an auditory notification, transmitting an alert to a fire alarm system, or disconnecting power to the cooktop or a heating element.

109. The method of claim 108, further comprising escalating the action in stages based on persistence or seventy of the predicted condition.

110. The method of claim 107, further comprising imaging the cooktop to obtain a cooktop image sequence, segmenting the cooktop image sequence into one or more burner-specific image sequences, and selecting a burner-specific image sequence as the burner image sequence.

111. The method of claim 107, wherein the trained model resides on the controller, in a remote computing resource accessible over a network, or is partitioned between the controller and the remote computing resource.

112. The method of claim 107, wherein the condition includes at least one of: fire, boil-over, operating status, pot-too-hot, uncovered burner, or a predicted temperature at the horizon time.

113. The method of claim 107, wherein initiating the action comprises comparing the generated output with a threshold that depends on an operating mode of the cooktop.

114. The method of claim 107, further comprising reconnecting power after the generated output indicates that a predicted unsafe state is no longer present.

115. The method of claim 107, wherein the burner images are captured at a constant frame rate and the input window advances at a timepoint frequency equal to or less than the frame rate.

116. The method of claim 107, further comprising normalizing the burner images prior to generating the output using dataset-level normalization parameters.

117. A system for assembling a cooktop-event prediction training dataset, comprising:• a memory storing sensor-derived cooktop image data; and• a processor configured to:■ provide a burner image sequence comprising a plurality of time-ordered burner images;■ identify a plurality of timepoints in the burner image sequence; and■ for each timepoint, assemble a timepoint training instance comprising an input window including burner images associated with a window time period and a cooktop-event label corresponding to a condition at a horizon time.

118. The system of claim 117, wherein the processor is further configured to perform at least one of: normalization, flipping, rotation, de-skewing, and noise injection on the burner image sequence.

119. The system of claim 117, wherein the memory further stores labels generated by manual labeling, automated labeling, or both.

120. The system of claim 117, wherein the burner images comprise thermal images, visible-light images, infrared images, or a combination thereof.

121. A system for generating a cooktop-event prediction model, comprising:• a memory storing a training dataset comprising a plurality of timepoint training instances, each instance including an input window and a corresponding cooktop-event label; and• a processor configured to:■ operate a machine learning model to produce one or more cooktop-event labels based on input windows; and■ modify parameters of the machine learning model based on a comparison between the predicted labels and the corresponding cooktop-event labels.

122. The system of claim 121, wherein the processor is further configured to partition the training dataset into training, validation, and testing subsets and to tune hyperparameters based on the validation subset.

123. The system of claim 121, wherein the machine learning model comprises a convolutional neural network, a recurrent neural network, a long short-term memory network,a transformer, a 3D-CNN, a support vector machine, a decision tree, a random forest, or an autoencoder.

124. A system for predicting a cooktop event at a cooktop, comprising:• at least one sensor configured to collect a burner image sequence comprising a plurality of time-ordered burner images;• a memory storing a trained cooktop-event prediction model; and• a controller configured to:■ maintain an input window comprising a most-recent sequence of burner images;■ generate, using the trained model, a cooktop-event classification or regression output corresponding to a horizon time; and■ initiate an action based on the generated output.

125. The system of claim 124, wherein the at least one sensor comprises a thermal imaging sensor and a visible-light imaging sensor.

126. The system of claim 124, wherein the action comprises issuing a visual or auditory notification, activating a fire alarm system, or disconnecting power to the cooktop or a heating element.

127. The system of claim 124, wherein the controller is further configured to image the cooktop, segment the cooktop image sequence into burner-specific image sequences, and select a burner-specific image sequence as the burner image sequence.

128. The system of claim 124, wherein the trained model is executed on the controller, on a remote computing resource, or is split between the controller and the remote computing resource.

129. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising the method of any of claims 76-96.

130. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising the method of any of claims 97-106.

131. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising the method of any of claims 107-116.

132. The method of claim 76, wherein the window time period corresponds to a time period of up to 60 seconds and the horizon time corresponds to a time period of up to 120 seconds.

133. The method of claim 76, wherein the window time period corresponds to a time period of between two to ten seconds and the horizon time corresponds to a time period of between five to sixty seconds.

134. The method of claim 76, wherein the segmentation of claim 78 employs a fixed spatial mapping to known burner positions.

135. The method of claim 107, wherein the regression output is a predicted maximum burner-surface temperature over the horizon time and the action is triggered when the predicted maximum exceeds a threshold.

136. The system of claim 124, wherein the controller escalates from a visual notification to an auditory notification and then to power disconnection when the predicted probability of fire remains above a threshold for a first time interval and a second, longer time interval, respectively.

Images