Monitoring and controlling a domestic cooking appliance, and domestic cooking appliance

Abstract

The invention relates to a method for monitoring and / or controlling a domestic cooking appliance (1) having a cooking chamber (3) and a cooking-chamber camera (6), in which method a sequence of images from the cooking chamber (3) is captured by means of the cooking-chamber camera (6), and a time-lapse video is generated from the sequence of captured images, the time-lapse dynamics (T(j), ∆fr) of said time-lapse video being varied. The invention also relates to a domestic cooking appliance (1) which is designed to carry out the method. The invention also relates to a system (1, 2) comprising a domestic cooking appliance (1) and an external display unit (2) which can be coupled to the domestic cooking appliance (1) for data communication, wherein the system (1, 2) is designed to carry out the method. The invention is particularly advantageously applicable to ovens.

Description

[0001] 202402302

[0002] 1 / 30

[0003] Monitoring and controlling a household cooking appliance as well as a household cooking appliance

[0004] The invention relates to a method for monitoring and / or controlling a household cooking appliance with a cooking chamber and a cooking chamber camera, in which a sequence of images from the cooking chamber is recorded by means of the cooking chamber camera. The invention also relates to a correspondingly equipped household cooking appliance and a system comprising such a household cooking appliance and an external display unit that can be connected to the household cooking appliance via data transmission. The invention is particularly advantageously applicable to ovens.

[0005] Cooking processes are generally lengthy, and their progress can only be accurately assessed by observing them over an extended period. Many cooking processes cannot be judged externally, or only with great difficulty; that is, there are no, or only barely perceptible, changes in the food being cooked that would allow conclusions to be drawn about its condition, such as its degree of doneness.

[0006] However, there are also a number of cooking processes that can be easily assessed externally. An example of this is any dish that undergoes a change in volume during preparation, particularly a regular increase in volume. This can include dough-based foods such as bread, pastries, cakes, flans, tarte flambée, pizza, and the like. These cooking processes generally proceed in three phases: in the first phase, little is observable, as the food is still raw and must first absorb heat. In the second phase, internal gas production occurs, resulting in a noticeable increase in volume. In the third phase, hardly any further increase in volume is observable, but noticeable browning often occurs. The cooking process is usually complete at this point.

[0007] The cooking process is therefore subject to dynamic changes that are generally not linear over time. The user usually doesn't know in advance exactly when to observe. An experienced observer can perform regular checks and draw appropriate conclusions. However, an inexperienced observer faces the problem that, while they can observe just as well, they lack the experience to assess how far along the cooking process might be. For example, 202402302

[0008] 2 / 30

[0009] unsure whether the bread will rise further or whether the observed browning is just the beginning and will increase significantly.

[0010] US 10 808941 B2 uses a camera to provide the user with images taken during the cooking process, along with images retrieved from a database, for comparison on their mobile device. This is intended to allow the user to determine whether certain cooking stages – as determined by comparing annotated reference images with images from the current cooking process – have already been reached. Only snapshots are compared at a time.

[0011] US 2023 / 0067540 Al describes the use of images of the food being cooked to provide placement guidance and advice on which cooking area of ​​the appliance and in which arrangement the food should best be placed.

[0012] US 11 506 395 B2 provides the user with a food classification system and recommended process settings. At the end of the cooking process, the user is asked to assess the cooking success, which in turn helps to improve the process settings for the next cooking process. The feedback is presented using various images from the cooking process, accompanied by corresponding time-based information.

[0013] DE 10 2019 216 682 A1 discloses a method for determining a target processing state of at least one food item to be cooked using a cooking appliance, wherein a user is provided with a set of images of the food item in different processing states for selection, with respective measurement signatures assigned to the images, and if a user selects one of the images, the cooking appliance adopts the measurement signature associated with the selected image as the target measurement signature. A further method serves to operate a cooking appliance, wherein a cooking process is carried out until a target processing state, adopted by means of the method for determining the target processing state, is reached. A cooking appliance has a cooking chamber, at least one sensor connected to the cooking chamber, and a data processing device, wherein the cooking appliance is configured to carry out the method.DE 10 2019 216 682 Al is particularly applicable to ovens with at least one cooking chamber camera, especially for determining or selecting a degree of browning of food.202402302.

[0014] 3 / 30

[0015] EP 2 930 918 Al discloses an oven comprising a muffle, equipment for heating an active zone defined within the volume delimited by the muffle, a camera for capturing one or more images showing food positioned in the active zone, and a transmission unit connected to the camera and configured to send the one or more images to an auxiliary device configured at least to display the one or more images.

[0016] US 2023 / 0067540 discloses an end device and a cooking appliance capable of controlling the arrangement of food within the cooking appliance, which comprises multiple heat sources to create multiple cooking zones with varying heating characteristics. The end device includes an imaging device, a user interface designed to display an image captured by the imaging device and to receive user input, and a controller configured to control the user interface, upon determining that the captured image corresponds to an image of a cooking appliance and upon receiving food information from the user, to display an augmented reality (AR) image of the food placed over a recommended cooking zone beneath multiple cooking zones of the cooking appliance.

[0017] WO 2016 / 179424 Al discloses an attached oven comprising a series of sensors in the cooking chamber and a processor configured to automatically identify food in the cooking chamber based on sensor measurements and to automatically control the heating element based on the identity of the food.

[0018] The object of the present invention is to overcome at least some of the disadvantages of the prior art and, in particular, to provide an improved means of monitoring and / or controlling a treatment process taking place in the cooking chamber, especially of monitoring the state of food being treated by means of a cooking process and controlling the treatment process depending on this state.

[0019] 4 / 30

[0020] This task is solved according to the characteristics of independent claims.

[0021] Preferred embodiments can be seen in particular in the dependent claims.

[0022] The task is solved by a method for monitoring and / or controlling a household cooking appliance with a cooking chamber and a cooking chamber camera, in which a sequence of images from the cooking chamber is recorded using the cooking chamber camera and a time-lapse video is generated from the sequence of recorded images, the time-lapse dynamics of which is varied or is.

[0023] This method advantageously allows the user to recognize changes in the food being cooked within a manageable timeframe, changes that would otherwise only occur over an extended period and are therefore not easily observed. For example, changes in the color, volume, and / or surface texture of cooked food are relatively easy to detect using the time-lapse effect, whereas comparative observation during a cooking process in real time is very difficult, if not impossible. This relieves the user of the need for continuous or even regular monitoring, allowing them to review the history and progress of the cooking process up to the current point at longer intervals by replaying the accumulated footage and assessing how far along the cooking process is. This also simplifies the automatic monitoring of the food being cooked.

[0024] A household cooking appliance can be, for example, a conventional oven, a microwave oven, a steam oven, or a combination thereof, such as an oven with microwave and / or steam oven functionality. The cooking chamber typically has a loading opening that can be closed by means of a closure device; for example, a door for a front loading opening and a lid for a top loading opening.

[0025] The oven camera is designed, i.e., installed and positioned, to take pictures from inside the oven, in particular of food being cooked inside. The oven camera can be mounted, for example, on the ceiling or on a side wall of the oven, or alternatively or additionally on the door / closing mechanism. The household oven can have one or more oven cameras.

[0026] 5 / 30

[0027] The oven camera can capture images in a time-lapse sequence. It can be sensitive to and record in the visible and / or infrared spectral ranges. It can be a digital camera, such as a color camera like an RGB camera and / or an IR camera. The oven camera can generate a time-lapse sequence of images captured sequentially. These images can be stored, for example, in the camera's own memory, the oven's memory, or an external device.

[0028] The sequence of captured images forms an initial image set {i} containing IGES images, which serves as the basis for generating the time-lapse video. The order of the images in the initial image set {i} corresponds specifically to the order in which they were captured. Each image can be assigned an image number i, which corresponds to the sequence of image capture. Depending on the requirements, the first image can be assigned the image number i = 0 or i = 1, where the image number IMAX of the last image in the sequence is then IMAX = (IGES - 1) or IMAX = IGES, respectively. In particular, the first image of the sequence of images, or the initial image set {i}, can be considered the beginning of a treatment process, especially a cooking process.

[0029] With a known sampling frequency or image acquisition rate of the cooking chamber camera, and if a specific image is equated with the start of the cooking process, the images or image numbers i of the initial image set {i} can be equated with a corresponding elapsed duration of the cooking process or the cooking time to date. The image with the number i then shows the state recorded at the corresponding cooking time, e.g., of the food being cooked. This is particularly easy to implement if the cooking chamber camera records images with a fixed sampling frequency or image acquisition rate. The image number i is then proportional to the elapsed cooking time by the inverse of the image acquisition rate. Consequently, the cooking time and the image number i can be used interchangeably.

[0030] It is a further development that the images of the initial image set {i} are captured with identical camera settings. In particular, the white balance (which is preset to active on many commercially available cameras) and / or exposure should advantageously be chosen so that the individual images combined into a time-lapse video have at least largely the same brightness and color, in order to achieve particularly high image quality.

[0031] 6 / 30

[0032] This can be ensured particularly easily with fixed exposure times (e.g., 0.05 s), ISO values ​​(e.g., ISO 400), and color temperature values ​​for white balance (e.g., 5000 K). These values ​​can advantageously be adjusted to the lighting conditions, for example, to the light intensity and color temperature of the incandescent or LED lighting used in the cooking appliance. It is also possible to achieve the most homogeneous color representation possible through subsequent image processing within the original image set {i}. This allows, for example, the correction of lighting disturbances that can enter the cooking chamber through the viewing window (such as kitchen lighting, sunlight, etc.).

[0033] To improve the visualization of the cooking process, users can be given a visual cue through color-coded displays during training. For example, when a desired degree of browning is reached, the corresponding area can be marked in signal colors. This makes the progress of the cooking process particularly clear and easy for the user to grasp.

[0034] It is a further development that all images captured by the oven camera are saved. It is a further development that only every nth image captured by the oven camera, where n > 1, is saved, to advantageously save storage space when the oven camera's sampling rate is higher than required. In particular, the saved image sequence then forms the initial image set {i}.

[0035] It is a further development that the time-lapse dynamics are varied by varying the step size of the previously recorded images stored in the data storage, i.e., by varying n. For example, at the beginning of a cooking process, every n-th image can be stored, and as the process progresses, every m-th image can be stored, and so on, where n-th > m. The initial set of images {i} can then be used as the time-lapse image set {j} and played back, for example, with a constant display duration or playback speed / frame rate. Alternatively, the recording rate over time can be varied, in particular, the recording frequency can be increased as the cooking process progresses. The time-lapse dynamics can therefore generally also be a [compared to a real] 202402302

[0036] 7 / 30

[0037] Timeline of the cooking process - include image-side dynamics, instead of or in addition to playback-side dynamics.

[0038] It is a design in which the time-lapse dynamic includes at least one display duration of at least one image to be displayed in the time-lapse video, i.e., the display duration is varied or dynamically adjusted during the course of the time-lapse video.

[0039] As the duration of the treatment process increases, the number of IGES images in the initial image set {i} typically increases, and thus the number of images from which the time-lapse video can be generated.

[0040] In this advanced training, the process involves regenerating a time-lapse video after a specific time period or after capturing a certain number of additional images since the previous time-lapse video was created. This can also be referred to as updating the time-lapse video. The advantage of this approach is that the time-lapse video is adapted to the duration of the cooking process and thus also includes the latest observations of the cooking chamber, particularly the food being cooked. For example, a new time-lapse video can be generated or the existing one updated every 10 seconds, 30 seconds, or 1 minute. With a constant frame rate of one frame per second (1 fps) for the cooking chamber camera, this corresponds to an update every 10, 30, or 60 captured images, respectively.

[0041] Monitoring a household cooking appliance includes, in particular, monitoring the cooking chamber and / or its contents during a treatment process. This treatment process can be, for example, a cooking cycle, also referred to as a cooking procedure or cooking process. The treatment process can also be a self-cleaning cycle, such as pyrolysis or catalysis.

[0042] Controlling the household cooking appliance can include, for example, changing a cooking parameter such as the cooking chamber temperature, switching to a different cooking phase, activating or deactivating a steam generator and / or a microwave generator, and / or ending the cooking process. Different cooking phases can be distinguished, for example, by different cooking chamber temperatures, different heating methods, different 202402302

[0043] 8 / 30

[0044] Differentiating between ventilation scenarios, varying humidity levels, etc., is possible. Controlling a household cooking appliance can also be described as controlling a cooking process.

[0045] A time-lapse video is, in particular, an image sequence generated from the stored image sequence that runs faster or is accelerated compared to the actual duration of the displayed treatment process; that is, it displays at least one section of the treatment process faster than real time. This also includes the case where at least one section of the time-lapse video runs in real time and therefore does not exhibit time-lapse, while at least another section of the time-lapse video runs in time-lapse or is accelerated. The time-lapse dynamics can be understood, in particular, as the magnitude or strength of the acceleration. The sequence of images of the time-lapse video forms a time-lapse image set {j} with JGES images of image number j, where, analogous to {i}, the sequence of image numbers can be, for example, j = 0, 1, ..., JMAX = QGES - 1) or j = 1, 2, ..., JMAX = JGES.The order of the images in the time-lapse image set {j} corresponds specifically to the order in which they are played back in the time-lapse video. The original image set {i} is thus mapped to the time-lapse image set {j}. In a further development where all images of the original image set {i} are played back in the time-lapse video, the original image set {i} corresponds to the time-lapse image set {j}. If only a subset of the images of the original image set {i} are played back in the time-lapse video, then {j} < {i} or Ji < Ji.

[0046] The time-lapse video can therefore be used to monitor and / or control a cooking process taking place in the cooking chamber. The time-lapse video can be made available to a user for monitoring / viewing or can be played back. This allows the user to better recognize changes visible from the outside than with a real-time video or individual still images and to control the cooking process accordingly. For example, during a cooking process, the user can stop the cooking process, change the cooking chamber temperature, initiate a steam burst, or handle the food, such as basting, etc. Alternatively or additionally, the cooking process can be automatically monitored using the time-lapse video, for example, by analyzing the images in the time-lapse video to determine the current state of the food, e.g., based on an externally observable, variable parameter ("change parameter") such as its volume, gloss level, and / or browning level, etc.to determine and / or to recognize the achievement of a target state. It has been shown that 202402302.

[0047] 9 / 30

[0048] Changes in parameters can often be detected more reliably and automatically using time-lapse video than with real-time video. Based on the detected state of the food being cooked, the household cooking appliance can control the cooking process, in particular ending the cooking process upon detecting that the target state has been reached, changing one or more treatment parameters, switching to a keep-warm phase, etc.

[0049] It is a further development that the time-lapse dynamics are varied depending on the position or image number of an image in the original image set {i} and / or in the time-lapse image set {j}. This advantageously allows for the implementation of a particularly useful acceleration in a relatively simple way.

[0050] One implementation involves varying the time-lapse dynamics by changing the acceleration of the sequence of images in the time-lapse video as these images progress within the recorded or saved sequence, or as the image number increases, and in particular by decreasing the acceleration at least in certain phases. For example, a time-lapse dynamic can be implemented that accelerates sharply at the beginning and end, but slows down in the middle.

[0051] It is a further development technique to vary the time-lapse dynamics by decreasing the acceleration of the image sequence as the images move up the sequence. This corresponds to a reduction in the time-lapse effect towards the end of the video. This technique takes advantage of the fact that, for many foods being monitored, the parameters being observed, such as volume and / or browning, change only slightly at the beginning of the cooking process, while these parameters change more significantly as the cooking progresses. This approach allows initial cooking phases with minor changes in the food being cooked to be processed more quickly, thus requiring less attention from the user and having less impact on automatic analysis, while simultaneously reducing processing time.

[0052] It is a further development feature that, after the time-lapse function has ended or the time-lapse video has finished, the system seamlessly switches to real-time playback as a live video. This advantageously allows for a particularly seamless transition from 202402302.

[0053] 10 / 30

[0054] The generated image sequence provides a live view of the food being cooked. This, in turn, allows the user to have a particularly immersive and intuitive overview of the cooking progress.

[0055] One implementation is that the time-lapse dynamics include at least one display duration for a single image in the time-lapse video, meaning the display duration is varied or dynamically adjusted throughout the video. This is advantageously easy to implement. For example, the display duration of images in a sequence can be increased as their position in the sequence progresses compared to the preceding image. For instance, the first image in the time-lapse video could be displayed for one second, and subsequent images could be assigned increasing display times until the final image (JMAX) reaches the total duration of the time-lapse video. A further development involves changing only the display duration of at least one image in the time-lapse video compared to the initial set of images {i}, while the number of images remains unchanged.This corresponds to the statement that the original image set {i} is transferred or mapped unchanged into the time-lapse image set {j}. It is a further development that the display duration of at least one image in the time-lapse video is changed compared to the original image set {i}, and that at least one image from the original image set {i} is not transferred to the time-lapse image set {j}.

[0056] It is a configuration such that the display duration T(j) of a j-th image in the time-lapse video according to Eq. (1)

[0057] T(D = (j CE s - f) a - b

[0058] JGES is the total number of images in the time-lapse video or in the time-lapse image set {j}, where a is a freely selectable exponent and b is a freely selectable, constant offset. T(j) can, for example, be specified in seconds. In this configuration, without further measures, the playback duration increases with increasing JGES. It is a further development that the images of the time-lapse video correspond to the images of the recorded or saved image sequence, i.e., {i} = {j}. An alternative development is that {j} < {i}. Eq. (1) includes the fact that JGES is correlated with the already elapsed duration IGA of the treatment process, in particular the cooking process, and the display duration T(j) is adjusted accordingly, in particular 202402302

[0059] 11 / 30

[0060] The time-lapse video is dynamically updated to reflect the elapsed duration of the treatment process. This, in turn, provides the user with a particularly smooth and dynamic visual impression of the treatment, especially the cooking process, and also enables particularly reliable automatic monitoring of the treatment.

[0061] One implementation is that the time-lapse dynamics include at least one playback speed of the time-lapse video, meaning the playback speed is varied or dynamically adjusted over the course of the video. The playback speed is conceptually inversely proportional to the display duration of an image; that is, a higher playback speed corresponds to a correspondingly shorter display duration. Therefore, the statements regarding display duration also apply analogously to the playback speed. However, setting a playback speed is easier to implement in practice than setting the display duration. Defining the playback speed via a continuously differentiable function is a particularly advantageous refinement for achieving the smoothest possible transitions.

[0062] One implementation involves the time-lapse dynamics including a step size in the sequence of captured images or images in the source image set {i} that are displayed or intended to be displayed in the time-lapse video. This offers the advantage that the display duration and playback speed of the time-lapse video can be kept constant while varying the time-lapse dynamics. This step size refers to the amount by which the position of the next image in the stored image sequence, included in the time-lapse video, changes relative to the preceding image. The step size Afr between two immediately consecutive images i and (i+1) of the source image set {i} is "1", the step size between an image i and the next image but one (i+2) is "2", and so on. In other words, (Afr - 1) corresponds to the number of skipped images.In this configuration, if the step size Afr = 1, this means that two immediately consecutive images from the source image set {i} have been included in the time-lapse video; if the step size Afr = 2, two images after the next one from the source image set {i} have been included in the time-lapse video, and so on. The time-lapse effect is greater the larger the step size Afr is. This configuration therefore means that not all images from the source image set {i} are included in the time-lapse video.

[0063] 12 / 30

[0064] but at least partially, one or more images from the source image set {i} are skipped or not considered between two images to be displayed consecutively in the time-lapse video. In particular, the step size or the number of skipped images can depend on the position of one or both images from the source image set {i} to be displayed consecutively in the time-lapse image set {j}. For example, if a cooking process with decreasing acceleration over time is to be played back in the time-lapse video, a larger step size Afr can be set at the beginning of the source image set {i} than later in the source image set {i}. This can also be expressed as the number of skipped images from the source image set {i} between two images to be displayed consecutively in the time-lapse video decreases as the position in the original image sequence and / or in the time-lapse video progresses.The step size does not need to be continuously reduced; it can be kept constant for more than two frames of the time-lapse video before being reduced later. This corresponds to a section-by-section reduction of the step size. A further development is that towards the end of the time-lapse video, the step size Afr = 1, or the number of frames skipped in the initial frame set {i} is zero, meaning no more frames from the initial frame set {i} are skipped. In this case, no time-lapse effect occurs.

[0065] It is a configuration in which the step size Afr is iterative or sequential according to GL (2)

[0066] Africa

[0067]

[0068] With IGES being the total number of images in the source image set {i}, i being the image number of the last image selected for inclusion in the time-lapse video, Atz being the duration of the time-lapse video, and m being a selectable parameter, where m is a freely selectable variable describing the course of the time compression, Afr(i) thus describes the step size from an image i of the source image set {i} included in the time-lapse image set {j} to the next image of the source image set {i} to be included in the time-lapse image set {j}. In other words, the mapping i -> j and (ii- Afr(i)) -> (j + 1)- holds. With respect to Eq. (2), a beginning of the 202402302

[0069] 13 / 30

[0070] Using an initial image set {i} with image number i = 0 is particularly advantageous, where image i = 0 is adopted as the first image in the time-lapse image set {j}. This formula results in a particularly high acceleration at the beginning of the time-lapse video, which is especially beneficial for cooking processes where practically no relevant change is observable at the beginning.

[0071] It is a configuration such that the step size Afr according to Eq. (2A)

[0072] m • (J-GES — i) i-GES — i

[0073] Afr (i) =

[0074] At ZÄ m At ZÄ • fps

[0075] The number of frames per second (frame rate) in the time-lapse is determined using fps. The first term corresponds to equation (2). The second term increases the range of values ​​for m within which, regardless of the frame rate (fps) of the time-lapse video, the frames in the time-lapse are played back up to the current stage of the cooking process (corresponding to the last frame JMAX in the time-lapse video).

[0076] It is a configuration such that the step size Afr according to Eq. (3)

[0077] Africa

[0078]

[0079] The formula is determined using IGES as the total number of images in the initial image set {i}, i as the image number of the last image selected for inclusion in the time-lapse video, Atz as the duration of the time-lapse video, takt as the current runtime of the time-lapse, and k as a freely selectable parameter. This formula results in a lower acceleration at the beginning of the time-lapse video than equation (2). The current runtime takt can be replaced by an equivalent term that uses the image number i corresponding to the current runtime takt as a variable.

[0080] It is a configuration such that the step size Afr according to Eq (3A)

[0081]

[0082] 202402302

[0083] 14 / 30

[0084] is determined. The first term corresponds to Eq. (3) and the second term to the second term from Eq. (2A).

[0085] This implementation maintains a constant playback duration (Atz) of the time-lapse video during the recording of the image sequence. The time-lapse video "during the recording of the image sequence" can also be interpreted as "during a treatment process." This approach also covers the scenario where the number of images displayed in the time-lapse video, or the time-lapse image set {j}, increases as the treatment progresses, since the total number of images in the initial image set {i} also increases. Specifically, the display duration T(j), the playback speed, and / or the step size Afr(i) can be adjusted to ensure that the playback duration (Atz) remains approximately the same even with updated time-lapse videos.For example, the first image in the time-lapse video can be displayed for one second, and subsequent images can be assigned increasing display times until the final image, JMAX, reaches the specified playback duration, Atz. In particular, if only the display duration T(j) is changed, {i} = {j} can hold true.

[0086] An increasing display time with a constant playing time Atz" can be achieved, for example, according to the following example, where the playing time Atz" of the time-lapse video is 8 s, and the time-lapse video comprises four images, as shown in Table 1:

[0087]

[0088] Table 1

[0089] From this calculation example, a suitable formula can be determined by means of a curve fit in the plot of T(j) against j, in this case, for example, T (j) = 0.1111 ■ i 3 - 0.5 ■ i 2+ 1.0556 ■ j + 0.3333. By normalizing and converting to the desired playing time Atz«, the respective 202402302 can then be calculated for any number of images j.

[0090] Display durations T(j) can be calculated, e.g. in Table 2 for 16 images with Atz« = 8 s:

[0091]

[0092] Table 2

[0093] It is a design feature where the duration "Atz" of the time-lapse video increases less than linearly with the number of images in the sequence. This means that the duration "Atz | m" of a time-lapse video with m images and the duration "Atz | n" of a (later) time-lapse video with n > m images satisfy the relationship "Atz | n / Atz | m < n / m". This less-than-linear increase can also be described as a sub-proportional increase. This has the advantage of preventing the time-lapse video from becoming too long for very long cooking times. For example, while a frozen pizza is ready after 10 to 15 minutes, a roast can take several hours to cook. A time-lapse duration of, say, 5 seconds for the pizza might still be appropriate.However, if this is extended linearly, the roasting video would stretch to a minute, which could tire the user and would also be longer than necessary for automatic evaluation.

[0094] It is a design feature such that the duration "Atz" of the time-lapse video increases less than linearly, or less than proportionally, with the increasing number of frames in the time-lapse video. For example, if the time-lapse video contains JGES = m frames at a certain time and JGES = 2 m frames at a later time, the duration "AtzR" of the time-lapse video at the later time is not twice as long, but less than twice as long.

[0095] 16 / 30

[0096] It is a configuration such that the playing time Atz" of the time-lapse video according to Eq. (4)

[0097]

[0098] The cooking time is determined using tGAR, the elapsed cooking time, and freely selectable parameters p and q. This equation (4) has the advantage that faster cooking processes are not shortened too much, e.g., compared to a playing time that is proportional to the square root of the recording time. On the other hand, a purely logarithmic relationship might lead to near-asymptotic behavior too early for typical cooking times. For p, in particular, p > 0, especially 0 < p < 10. For q, in particular, q < 1, especially 0.2 < q < 1. It is a further development that the linear parameter p is user-modifiable to allow the user to control the video length.

[0099] This configuration involves determining a gradient of at least one monitored variable ("change gradient") based on the sequence of recorded images or the sequence of images in the initial image set {i}, and then determining the time-lapse dynamics based on this change gradient. This offers the advantage of allowing the time-lapse dynamics to be specifically adapted to the rate of change of the variable during the treatment process. In particular, the acceleration of the time-lapse can be set lower in those time periods of the treatment process in which the change gradient is high (i.e., it changes rapidly), and higher in those time periods in the treatment process in which the change gradient is low (i.e., it changes slowly or practically not at all).For example, with a high rate of change, the display duration can be set to a correspondingly long time and / or the step size to a correspondingly short time. The rate of change can be determined, for example, using recognition algorithms or artificial intelligence.

[0100] It is a configuration in which at least one change parameter of the food being cooked is monitored during the cooking process, and the time-lapse dynamics are adjusted depending on the type of food being cooked. This has the advantage that the acceleration of the time-lapse video depends on a 202402302

[0101] 17 / 30

[0102] The time-lapse dynamics can be adjusted to reflect the qualitatively known development of at least one variable in the food being cooked. For example, the time-lapse dynamics can be adapted to typical developments of at least one variable in the food being cooked, such as the typical duration of a noticeable change phase (e.g., how quickly the food rises significantly and / or browns) and / or the position of such a change phase during the cooking process (e.g., in which cooking phase the food rises and / or browns).

[0103] A simple gradient of change can be derived, for example, by transforming the image points or pixels corresponding to the food being cooked into grayscale values ​​ranging from 0 (black) to 255 (white). From this, an average grayscale value for the food being cooked can be calculated. This value decreases as browning progresses and, when plotted against the cooking time, can be used as a parameter for time-lapse dynamics based on the curve's slope.

[0104] The problem is also solved by a household cooking appliance comprising a cooking chamber, a cooking chamber camera, and a device for generating a time-lapse video, wherein the household cooking appliance is configured to perform the method as described above. The household cooking appliance can be designed analogously to the method, and vice versa, and offers the same advantages. The device for generating the time-lapse video can be a data processing device, in particular one that also includes a data storage device for saving the images captured by the cooking chamber camera.

[0105] It is considered further development that a display device is integrated into the household cooking appliance, on which the time-lapse video can be played. The display device can be a touch-sensitive or a non-touch-sensitive screen. The screen can be a color screen. The screen can be integrated into a control panel, in particular a control panel, of the household cooking appliance.

[0106] The household cooking appliance may have at least one communication module for data communication with external entities and, in particular, be configured to transmit time-lapse videos to at least one external entity via the communication module. The communication module may be a module for 202402302

[0107] 18 / 30

[0108] wired communication, e.g. an Ethernet module, and / or a module for wireless communication, e.g. a Bluetooth or WiFi module.

[0109] The external instance can be, for example, a network server, a cloud computer or cloud storage, and / or a user device, particularly a mobile user device. The user device is equipped for data communication with the household appliance, especially with a corresponding application program ("app"). The user device can be, for example, a desktop computer. The mobile user device can be, for example, a smartphone, a tablet PC, a laptop computer, etc. The user device can, in particular, serve as a display unit for showing a transmitted time-lapse video and therefore have a screen or be connected to a screen. The user device can be configured, for example, via the application program, to control the household appliance, for example, in the sense of a remote control.

[0110] The task can also be solved by a system comprising a household cooking appliance with a cooking chamber and a cooking chamber camera, and an external display unit that can be connected to the household cooking appliance via data transmission. The system is configured to carry out the procedure as described above. This offers the advantage that a user can monitor and / or control a treatment process, particularly the cooking process, remotely. The system can be configured analogously to the procedure, the household cooking appliance, and / or the external device as described above, and vice versa, and offers the same advantages.

[0111] The properties, features and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more easily understood in connection with the following schematic description of an exemplary embodiment, which will be explained in more detail in conjunction with the drawings.

[0112] Fig. 1 shows a sketch of a system consisting of a household cooking appliance and a mobile user device as a sectional view in side view;

[0113] Fig. 2 shows a family of curves with different display durations per image for the same initial image quantity; 202402302

[0114] 19 / 30

[0115] Fig. 3 shows a family of curves with different display durations per image for different initial image quantities and the same playing time of the time-lapse video;

[0116] Fig. 4 shows a plot of the playing time of a time-lapse video against an already elapsed cooking time; and

[0117] Fig. 5 shows a plot of the step size for setting up a time-lapse video against the playing time of the time-lapse video.

[0118] Fig. 1 shows a cross-sectional sketch of a system consisting of a household cooking appliance 1 in the form of an oven with or without additional functionality and a mobile user device 2 in the form of a smartphone that can be connected to it. The household cooking appliance 1 has a heated cooking chamber 3, the front loading opening of which can be closed by means of a door 4. The food to be cooked, G, in this case a baguette, is placed in the cooking chamber 3. A cooking chamber camera 6 is mounted on the ceiling of a cooking chamber wall 5 surrounding the cooking chamber 3 and is directed into the cooking chamber 3 from above or at an angle. This camera captures images from inside the cooking chamber 3 and has a field of view in which the food G is typically located.

[0119] Images captured by the oven camera – typically at a constant frame rate – are stored in a data storage device 7. The data storage device 7 can be part of a data processing unit, which here corresponds, for example, to the control unit 8 of the household appliance 1. The control unit 8 can generate a time-lapse video with varying time-lapse dynamics from the stored sequence of images. The sequence of captured images stored in the data storage device 7 can remain unchanged or can be converted into an image sequence tailored to the time-lapse video, e.g., with a variable step size when selecting the images for the time-lapse video.

[0120] The control unit 8 is connected to an internal screen 9 on which the time-lapse video can be displayed. The control unit 8 is also connected to a communication module 10, through which the control unit 8 can communicate with the mobile user device 2. In one variant, the images of the time-lapse video can be transferred to the user device 2 and played back on its screen. Alternatively, the sequence of recorded images can be displayed on 202402302

[0121] 20 / 30

[0122] The data will be transferred to user device 2, where a time-lapse video will be created.

[0123] The recording of images by the oven camera 6 can be started by a user, e.g., at the push of a button. Alternatively or additionally, the image sequence underlying the time-lapse video can be started automatically by the household oven 1 itself, in particular its control unit 8, for example, when the presence of food G in the oven chamber 3 has been detected by evaluating previously recorded images from the oven camera 6, or when the cooking process is started.

[0124] Fig. 2 shows a family of curves Kl to K5 representing display durations T(j) per frame j in the time-lapse video, with the same time-lapse frame set {j} and jcEs = 20 frames of the sequence j = 1, ..., JMAX = JGES. The time-lapse frame set {j} can, in particular, correspond to the original frame set {i}. The curves Kl to K5 were calculated according to Eq. (1) using the following parameters shown in Table 3:

[0125]

[0126] Table 3

[0127] The value of a is therefore always less than zero, while b > 0. The larger the absolute value of a, the more the display duration T(j) increases relative to the end of the curve. The larger the value of b, the shorter the display duration T(j) becomes for all i. For the same value of b, the display duration T(j) of the last frame j = JMAX is the same. This generally results in a different playback duration AtzR of the time-lapse video for different values ​​of a and b. The playback duration Atz" is shorter with a larger absolute value of a than with a smaller absolute value of a. The playback duration Atz" also decreases with a larger value of b. It is 202402302

[0128] 21 / 30

[0129] a further development that -1.0 < a < -0.7 holds, b is advantageously chosen such that the term value remains greater than 0 for all j.

[0130] Fig. 3 shows a family of curves representing different display durations T(j) per image j for different time-lapse image sets {j} and the same playback duration Atz« = 8 s of the time-lapse video. The time-lapse image set {j} can, in particular, correspond to the original image set {i}.

[0131] Five graphs with JGES = 4, 12, 20, 40, and 80 frames are considered here. The display durations T(j) for JGES = 4 correspond to the values ​​listed above in Table 1; the display durations T(j) for the other values ​​of JGES are derived from these, e.g., by curve fitting and subsequent normalization.

[0132] The higher the total number of images j displayed in the time-lapse video (JGES), the shorter the display durations T(j) per image j become. To facilitate monitoring and / or automatic evaluation, it can be advantageous, especially for larger time-lapse image sets {j}, to define a step size Afr(i) > 1. This means that only a portion of the images from the original image set {i} are displayed in the time-lapse video, thus reducing the total number of images JGES in the time-lapse image set {j}. In a particularly simple implementation, only every xth image with x = constant from the original image set {i} can be included in the time-lapse image set {j}. This corresponds to a constant step size Afr = (x - 1) with x = 2, 3, ... Alternatively, the step size Afr can decrease as the image number i increases, resulting in a reduction of the time-lapse effect towards the end of the video.

[0133] Fig. 4 shows a plot of the playing time Atz of the time-lapse video in seconds against the elapsed cooking time IGAR in seconds. Eq. (4) is used to calculate the playing time Atz, here using the parameters p = 6.5 and q = 0.33 as an example. The playing time Atz increases less than linearly with increasing cooking time IGAR. In the illustrated embodiment, the playing time AtzR after slightly over 7000 seconds is only about twice as long as it was shortly after the start of the cooking process.

[0134] Fig. 5 shows a plot of a step size Afr (i) for setting up a time-lapse video against a playing time AtzR of the time-lapse video. The plot is 202402302

[0135] 22 / 30

[0136] The curve VI calculated according to Eq. (2A) and the curve V2 calculated according to Eq. (3A). The playing time AtzR is the same for both curves VI and V2, since the same frame rate is assumed. Curve VI shows very strong time compression, i.e., that images recorded at the beginning of the cooking process are very shortened in the time-lapse video, while the time-lapse effect decreases significantly towards the end of the cooking process.

[0137] For both processes VI and V2, the total number of images in the initial image set {i} is assumed to be IGES = 1800. In this example, the cooking process is intended to last 1800 seconds, with one image being captured every second. The image number i thus corresponds to the state of the food after a running time of i seconds. Since the first image has the image number i = 0, the maximum image number IMAX = (IGES - 1) = 1799. The playback rate of the time-lapse video is fps = 10, meaning that the display duration T(j) of images j in the time-lapse video is a constant 0.1 s. The playback duration Atz" of the time-lapse video is 10.3 s. This is directly evident from the plot in Fig. 4. The parameter m, which describes the time compression curve, was set to 1.6 here to match the running time / playback duration Atz" and the playback rate fps.

[0138] The calculation of the images i to be transferred from the initial image set {i} to the time-lapse video or the time-lapse image set {j} is performed iteratively, row by row. To create the time-lapse video, the first image i = 0 from the initial image set {i}, which was captured at the beginning of the cooking process, is chosen as its first image j = 0. This corresponds to a mapping (i = 0) -> (j = 0).

[0139] The next image in the time-lapse video with image number j = 1, which is displayed 0.1 s later, corresponds (rounded to whole numbers) to the image (i + Afr) in the initial image set {i} with the step size relative to the previous image i = 0 according to Eq. (2A)

[0140] 1.6 • (1800 - 0) 1800 - 0

[0141] Afr = 69.0 + 17.4 = 86

[0142] 10, 3 1 - 6 + 10.3 • 10

[0143] Accordingly, the next image displayed is j = 1, which is image i = 86 from the initial image set {i}, within the time period 0.1 s to 0.2 s of the time-lapse video. 202402302

[0144] 23 / 30

[0145] This corresponds to a mapping (i = 86) -> (j = 1). The image i = 86 represents the state of the food after a cooking time of i / fps, where fps is the image acquisition rate of the cooking chamber camera 6, in this example after a cooking time of 86 seconds.

[0146] The step size Afr of the next selected image (i + Afr) refers to the last selected image with i = 86 and is calculated from Eq. (2A) to Afr = 82. Thus, for the period from 0.2 s to 0.3 s in the time-lapse, the image number i = 86 + 82 = 168 is shown. This corresponds to a mapping (i = 168) -> (j = 2).

[0147] These iterations continue throughout the duration of the time-lapse video:

[0148]

[0149] Table 4

[0150] Process V2 is less compressed. For example, k = 1.4 is assumed here. Then, under iterative or sequential calculation of the step size according to Eq. (3A) starting from the first image i = 0 of the initial image set {i}, the following applies:

[0151] 1.4 • (1800 - 0) 1800 - 0

[0152]

[0153] where the cooking time elapsed at the time of the first image i = 0 is 0 s. 202402302

[0154] 24 / 30

[0155]

[0156] Table 5

[0157] It is a further development that the time-lapse can be played back automatically. It is a further development that a user can view the video sequence by manually scrolling through it via a swipe gesture, for example, on a touchscreen of the household cooking appliance and / or an external device such as a smartphone. It is also possible to switch back to a linear timescale so that the finger's position on the screen corresponds to the linearly progressing cooking process on the time axis. It is a further development that the time-lapse can be played back in an endless loop.

[0158] Of course, the present invention is not limited to the embodiment shown.

[0159] In general, "ein", "eine", etc. can be understood to mean singular or plural, especially in the sense of "at least one" or "one or more", etc., unless this is explicitly excluded, e.g. by the expression "exactly one", etc.

[0160] A numerical value can also include the exact number specified as well as a standard tolerance range, unless explicitly excluded. 202402302

[0161] 25 / 30

[0162] Reference symbol list

[0163] 1 household cooking appliance

[0164] 2 User terminal

[0165] 3 Cooking chamber

[0166] 4-door

[0167] 5 Oven chamber wall

[0168] 6 Oven camera

[0169] 7 Data storage

[0170] 8 Control unit

[0171] 9 screen

[0172] 10 Communication module

[0173] G Cooking food

[0174] j Image number of an image j in the time-lapse video

[0175] K1-K5 progressions

[0176] T(j) Display duration of an image with image number j in the time-lapse video tGAR Elapsed cooking time

[0177] V1-V2 progressions

[0178] Afr step size

[0179] AtzR playing time

Claims

202402302 26 / 30 Patent claims 1. Method for monitoring and / or controlling a household cooking appliance (1) with a cooking chamber (3) and a cooking chamber camera (6), wherein - a sequence of images from the cooking chamber (3) is recorded using the cooking chamber camera (6) and - a time-lapse video is created from the sequence of recorded images, the time-lapse dynamics of which (T(j), Afr) is varied.

2. Method according to claim 1, wherein the time-lapse video is used to monitor and / or control a treatment process taking place in the cooking chamber (3).

3. Method according to one of the preceding claims, wherein the time-lapse dynamics (T(j), Afr) is varied by decreasing the acceleration of the sequence of images (j) in the time-lapse video as the position of these images in the sequence of recorded images progresses.

4. Method according to one of the preceding claims, wherein the time-lapse dynamics includes at least one parameter from the group - Display duration (T(j)) of the images (j) to be displayed in the time-lapse video, - Playback speed of the time-lapse video, - Step size (Afr) in the sequence of recorded images displayed in the time-lapse video, includes.

5. Method according to claim 4, wherein a display duration (T(j)) of a j-th image (j) in the time-lapse video according to T(j) = jGEs - i - b The total number of images in the time-lapse video is determined using JGES, where a is an exponent and b is a constant offset. 202402302 27 / 30 6. Method according to one of claims 4 to 5, wherein the step size (Afr) is changed with progressive position in the sequence of recorded images, in particular at least phaselessly reduced.

7. Method according to claim 6, wherein the step size Afr according to or according to m ' (J-GES 0 i-GES i Afr (i) = At ZÄ m At ZÄ • fps with IGES the total number of images in the initial image set {i}, i the image number of the last image selected for inclusion in the time-lapse video, Atz« the duration of the time-lapse video, fps the number of images per second in the time-lapse and m a selectable parameter.

8. Method according to claim 6, wherein the step size Afr according to & & or according to & The total number of images in the initial image set {i} is determined using IGES, i is the image number of the last image selected for inclusion in the time-lapse video, Atz is the duration of the time-lapse video, fps is the number of frames per second in the time-lapse, takt is the current runtime of the time-lapse, and k is a selectable parameter. 28 / 30 9. Method according to one of the preceding claims, wherein a playing time (Atz«) of the time-lapse video is kept constant during a recording of the sequence of images by the cooking chamber camera (6).

10. Method according to any one of claims 1 to 8, wherein the playing time (Atz«) of the time-lapse video is increased less than linearly with increasing number of images (j) in the time-lapse video.

11. Method according to claim 10, wherein the playing time Atz" of the time-lapse video according to is determined using tGAR, the cooking time elapsed so far, and p and q, which are freely selectable parameters.

12. Method according to one of the preceding claims, wherein a change gradient of at least one monitored change parameter is determined based on the sequence of recorded images and the time-lapse dynamics (T(j), Afr) is determined based on the change gradient.

13. Method according to one of the preceding claims, wherein at least one change parameter of a product (G) is monitored during a cooking process and the time-lapse dynamics (T(j), Afr) are adjusted depending on a type of product (G) being treated during the cooking process.

14. Household cooking appliance (1) comprising a cooking chamber (2), a cooking chamber camera (6) and a device for generating a time-lapse video, wherein the household cooking appliance (1) is configured to perform the method according to one of the preceding claims.

15. System (1, 2) comprising a household cooking appliance (1) with a cooking chamber (3) and a cooking chamber camera (6) and a household cooking appliance (1)202402302 29 I 30 external display unit (2) that can be connected via data technology, wherein the system (1, 2) is configured to carry out the method according to one of claims 1 to 13.

Images