Cooking apparatus and control method thereof

Abstract

A cooking apparatus is disclosed. The present cooking apparatus comprises a camera for capturing a cooking chamber, a memory, and a processor for: on the basis of cooking food inside the cooking chamber in response to operation information set by a user input, acquiring a first image by capturing the cooking chamber through the camera; identifying a type of the food on the basis of the first image; generating a prompt for requesting an expected image of the food at a cooking completion time, on the basis of the operation information and at least one of the first image and the type of the food; acquiring a second image by inputting the prompt to a first model; acquiring a third image by capturing the cooking chamber through the camera; and controlling an operation of the cooking apparatus on the basis of the second image and the third image.

Description

Cooking device and control method thereof

[0001] The present disclosure relates to a cooking device and a method for controlling the same, and more specifically, to a cooking device and a method for controlling the same that control the cooking environment based on a captured image of a food item.

[0002] A cooking device is a device configured to cook food. For example, a cooking device may include an oven, induction cooktop, electric range, gas range, etc. A service is being developed that provides an image of the cooking result by utilizing a camera equipped on the cooking device to capture an image of the food being cooked.

[0003] Meanwhile, conventionally, there was a problem in that it failed to reflect the need to more actively control the cooking device based on an image of the cooking result.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment, a cooking device including a cooking chamber may be provided. The cooking device may include at least one processor including a memory and processing circuitry for storing camera instructions for photographing the cooking chamber. The instructions may be executed individually or collectively by the at least one processor, based on the cooking device cooking food inside the cooking chamber in response to operation information set by user input, by photographing the cooking chamber through the camera to obtain a first image, identifying the type of food based on the first image, generating a prompt to request an expected image of the time when the cooking of the food is completed based on at least one of the first image and the type of food and the operation information, inputting the prompt into a first model to obtain a second image, photographing the cooking chamber through the camera to obtain a third image, and controlling the operation of the cooking device based on the second image and the third image.

[0006] According to one embodiment, a method for controlling a cooking device including a cooking chamber may be provided. The control method may include the steps of: acquiring a first image by photographing the cooking chamber through a camera included in the cooking device, based on cooking food inside the cooking chamber in response to operation information set by user input; identifying the type of food based on the first image; generating a prompt to request an expected image of the time when the cooking of the food is completed based on at least one of the first image and the type of food and the operation information; acquiring a second image by inputting the prompt into a first model; acquiring a third image by photographing the cooking chamber through the camera; and controlling the operation of the cooking device based on the second image and the third image.

[0007] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0008] FIG. 1 is a schematic diagram showing a cooking device according to one embodiment.

[0009] FIG. 2 is a block diagram illustrating the configuration of a cooking device according to one embodiment.

[0010] FIG. 3 is a block diagram illustrating the detailed configuration of a cooking device according to one embodiment.

[0011] FIG. 4 is a flowchart illustrating a method for controlling operation information of a cooking device based on an image of a cooking chamber according to one embodiment.

[0012] FIG. 5 is a diagram illustrating the operation of a cooking device generating a prompt according to one embodiment.

[0013] FIG. 6 is a flowchart illustrating the operation of a cooking device when the similarity is greater than or equal to a preset value, according to one embodiment.

[0014] FIG. 7 is a flowchart illustrating the operation of a cooking device when the similarity is greater than or equal to a preset value, according to one embodiment.

[0015] FIG. 8 is a flowchart illustrating the operation of a cooking device when the similarity is smaller than a preset value, according to one embodiment.

[0016] FIG. 9 is a drawing illustrating a display screen according to the operation of a cooking device according to one embodiment.

[0017] FIG. 10 is a flowchart illustrating a method for controlling operation information of a cooking device based on a thermal image of a cooking chamber taken according to one embodiment.

[0018] The terms used in the embodiments of this disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0019] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, operations, or components such as parts) and do not exclude the presence of additional features.

[0020] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0021] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0022] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0023] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0024] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor, except for a "module" or "part" that needs to be implemented in specific hardware.

[0025] In this specification, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0026] An embodiment of the present disclosure will be described in more detail below with reference to the attached drawings.

[0027] FIG. 1 is a schematic diagram showing a cooking device according to one embodiment.

[0028] According to one embodiment, the cooking device (100) may include a cooking chamber (10) and a camera (120). The cooking device (100) is a device that performs various cooking operations by applying high-temperature heat generated using electricity or gas to a food item, and may be an oven, induction cooker, electric range, or gas range, but is not limited to a specific type.

[0029] According to one embodiment, the cooking device (100) can cook food based on user input related to the operation of the cooking device. The user can place food to be cooked into the cooking chamber (10) and input operation information (e.g., cooking mode, operating temperature, operating time). The cooking device (100) can cook food according to the operating temperature, operating time, and cooking mode corresponding to the operation information.

[0030] For example, the operating temperature may refer to the temperature inside the cooking device (100). The cooking device (100) may maintain the internal temperature at the operating temperature using a heater mounted on the cooking device. For example, the operating time may refer to the time from when the cooking device (100) starts operating until when it ends. The cooking device (100) may end the operation of the cooking device (100) based on the fact that the operating time has been reached. For example, the cooking mode may refer to the cooking style of the food. The cooking device (100) may determine the method of heat transfer or the method of controlling internal airflow according to the cooking mode. The cooking mode may include, for example, convection, bake, variable broil, steam bake, steam roast, convection vegetable, air fry, and air sous vide.

[0031] The cooking chamber (10) includes a space for accommodating food and may be a space for performing cooking operations on the food. The cooking chamber (10) may include a heating unit for generating heat and a driving device capable of rotating and moving the food.

[0032] According to one embodiment, the cooking device (100) can photograph food located in the cooking chamber (10) using a camera (120). The camera (120) may be positioned to photograph the interior of the cooking chamber (10). For example, the camera (120) may be positioned to photograph the food being cooked from the front. Accordingly, the image obtained through the camera (120) may be a top-view image or an image taken at an angle close to a top-view image.

[0033] FIG. 2 is a block diagram illustrating the configuration of a cooking device according to one embodiment.

[0034] Referring to FIG. 2, the cooking device (100) may include a processor (110) (hereinafter referred to as processor (110)), at least one camera (120) (hereinafter referred to as camera (120)), and at least one memory (130) (hereinafter referred to as memory (130)).

[0035] The processor (110) can control the overall operation of the cooking device (100). Specifically, the processor (110) is connected to each component of the cooking device (100) to control the overall operation of the cooking device (100). The processor (110) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. The processor (110) can control one or any combination of other components of the cooking device (100) and can perform operations or data processing related to communication. The processor (110) can execute one or more programs or instructions stored in the memory (130) of the cooking device (100). For example, the processor (110) can perform the method according to one embodiment of the present disclosure by executing one or more instructions stored in memory (130).

[0036] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0037] The processor (110) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When the processor (110) is implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. Additionally, each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0038] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0039] In the embodiments of the present disclosure, a processor may mean a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto.

[0040] A camera (120) is provided inside a cooking room (10) and can photograph the inside of the cooking room (10). The camera (120) can photograph food contained inside the cooking room (10). The camera (120) can acquire an image corresponding to a subject by converting light emitted or reflected and transmitted from the subject into an electrical signal using an image sensor. According to one embodiment, the image sensor may include, for example, one image sensor selected from image sensors with different attributes such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes.

[0041] The memory (130) can store data necessary for various embodiments. Depending on the purpose of data storage, the memory (130) may be implemented in the form of a memory embedded in the cooking device (100) or in the form of a memory that can be attached to and detached from the cooking device (100). For example, data for operating the cooking device (100) may be stored in a memory embedded in the cooking device (100), and data for the expansion function of the cooking device (100) may be stored in a memory that can be attached to and detached from the cooking device (100). Meanwhile, the memory embedded in the cooking device (100) may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), etc.), non-volatile memory (e.g., OTPROM (one-time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash, etc.), hard drive, or solid state drive (SSD). Additionally, the memory that is detachable from the cooking device (100) may be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory connectable to a USB port (e.g., USB memory), etc. there is.

[0042] According to one example, the memory (130) may store at least one instruction or a computer program including instructions for controlling the cooking device (100).

[0043] In the above-described embodiment, various data is described as being stored in the external memory (130) of the processor (110), but at least some of the above-described data may be stored in the internal memory of the processor (110) according to at least one implementation example of the cooking device (100) or the processor (110).

[0044] FIG. 3 is a block diagram illustrating the detailed configuration of a cooking device according to one embodiment.

[0045] Referring to FIG. 3, the cooking device (100) may include a processor (110), a camera (120), a memory (130), an interface (140), and a heating unit (180). However, such a configuration is exemplary, and it is understood that in carrying out the present disclosure, new configurations may be added or some configurations may be omitted in addition to such configurations. Meanwhile, detailed descriptions of configurations shown in FIG. 3 that overlap with configurations shown in FIG. 2 will be omitted.

[0046] An interface (140) is a configuration created to interact between two or more systems, devices, programs, or users. The interface (140) may include at least one of a communication interface (150) and an input / output interface (160, 170).

[0047] The communication interface (150) includes a circuitry and can communicate with an external device (e.g., a server device and / or an external device). The processor (110) can receive various data or information from an external device connected through the communication interface (150) and can transmit various data or information to the external device.

[0048] The communication interface (150) can communicate with an external device through a nearby access point (AP). The access point (AP) can connect the local network (LAN) to which the cooking device (100) is connected to a wide area network (WAN) to which the external device is connected. The cooking device (100) can be connected to the external device through the network (WAN). Additionally, the communication interface (150) can perform device-to-device (D2D) communication with the external device. For example, the communication interface (150) can communicate with the external device over short distances without using an access point.

[0049] The communication interface (150) can communicate with an external device using various types of communication methods. For example, the communication interface (150) may include a LAN communication module such as an Ethernet module. The communication interface (150) may include wireless communication modules such as Wi-Fi, Wi-Fi Direct, Bluetooth, BLE (Bluetooth Low Energy), Zigbee, NFC, Z-Wave, and infrared communication. The communication interface (150) may include cellular communication modules such as 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), and 5G. The communication interface (150) may include communication modules such as HDMI (High-Definition Multimedia Interface) and USB (Universal Serial Bus).

[0050] The input interface (160) may include a circuit. The input interface (160) may receive user input for controlling the operation of the cooking device (100) and provide the user input to the processor (110). The user input may include an input for turning on / off the heating unit (180) of the cooking device (100), an input for setting the cooking temperature, an input for setting the cooking time, and an input for setting the cooking mode. The input interface may include, for example, a tact switch, a push switch, a slide switch, a toggle switch, a micro switch, a touch switch, a touch pad, a touch screen, a jog dial, and / or a microphone.

[0051] The input interface (160) may be provided in an area of ​​the outer wall surface of the cooking device (100). For example, the input interface (160) may be provided in an area of ​​the front portion of the housing. However, the present disclosure is not limited thereto, and the input interface (160) may be provided in various locations, such as the front and / or side of the housing.

[0052] The output interface (170) includes a circuit, and the processor (110) can visually or audibly convey various information related to the cooking device (100) to the user through the output interface (170). The output interface (170) may include a Liquid Crystal Display (LCD) panel, a Light Emitting Diode (LED) panel, a speaker, etc.

[0053] The heating unit (180) may be configured to heat the temperature inside the cooking chamber (10) to a cooking temperature set by the user. For example, a processor (110) may be electrically connected to the heating unit (180) and control the operation of the heating unit (180) based on operation information. The heating unit (180) may include an upper heating element, a lower heating element, a convection heating element, a grill heating element, etc. The location and number of heating units (180) may be determined in various ways.

[0054] FIG. 4 is a flowchart illustrating a method for controlling operation information of a cooking device based on an image of a cooking chamber according to one embodiment.

[0055] The processor (110) of the cooking device (100) can perform at least one of the operations of FIG. 4. When the instructions stored in the memory (130) of the cooking device (100) are executed by the processor (110) of the cooking device (100), the cooking device (100) can perform the operations of FIG. 4.

[0056] According to one embodiment, in operation 410, the processor (110) can heat food inside the cooking chamber based on operation information set by user input. For example, the processor (110) can heat food inside the cooking chamber in response to operation information set by user input. Alternatively, the processor (110) can cook food inside the cooking chamber in response to operation information set by user input.

[0057] For example, when the processor (110) receives user input to operate the cooking device (100), it can turn on the heating unit (180) for heating the cooking chamber (10).

[0058] According to one embodiment, the operation information may include information for controlling the heating unit (180) of the processor (110). For example, the operation information may include a cooking time, a cooking temperature, a cooking completion time, and a cooking mode. The cooking device (100) can heat food inside the cooking chamber (10) by controlling the heating unit (180) based on the operation information.

[0059] For example, if the user sets the cooking time (e.g., 1 hour), the cooking temperature (e.g., 180 degrees), and the cooking mode (convection), the processor (110) can control the cooking device (100) to maintain the temperature inside the cooking chamber (10) at 180 degrees for 1 hour using a convection heating element. Here, the cooking completion time may refer to the time when the cooking time has elapsed from the time cooking started.

[0060] According to one embodiment, when an operation to heat food inside a cooking chamber is started based on operation information set by user input in operation 420, the processor (110) can obtain an image (hereinafter referred to as the first image) by taking a picture of the cooking chamber (10) using a camera (120). For example, the first image may be a top view image of food located in the cooking chamber (10).

[0061] According to one embodiment, in operation 430, the processor (110) can identify the type of food based on the first image. For example, the processor (110) can identify the type of food by inputting the first image into a classifier model (hereinafter referred to as the second artificial intelligence model). The second artificial intelligence model may include a deep learning model trained to identify the type of food contained in the image. Meanwhile, in the present disclosure, the term 'artificial intelligence model' may be otherwise referred to as 'model'.

[0062] For example, the second artificial intelligence model may be implemented in the form of an on-device device included in the cooking device (100). The cooking device (100) may input a prompt into the second artificial intelligence model stored in memory (130) to obtain the type of food from the artificial intelligence model.

[0063] The processor (110) can obtain features of the first image through a convolutional neural network (CNN) included in the second artificial intelligence model. Additionally, the processor (110) can identify the type of food based on the features of the first image using the second artificial intelligence model.

[0064] According to one embodiment, in operation 440, the processor (110) may generate a prompt based on a first image, a type of food, and operation information. According to one embodiment, the processor (110) may generate a prompt to request an expected image of the time when the food is finished cooking based on the first image, a type of food, and operation information. The processor (110) may generate the prompt in a rule-based manner. According to one embodiment, the processor (110) may generate a prompt to request an expected image of the time when the food is finished cooking based on at least one of the first image and a type of food and operation information.

[0065] For example, the processor (110) can generate a prompt requesting the creation of an image related to a notification by inserting keywords (e.g., cooking time, cooking temperature, cooking mode, and type of food) into placeholders of a prompt template. For example, a placeholder may be a location in the prompt template where keywords are inserted. Placeholders to which keywords are mapped may be pre-specified according to the attributes of the keywords. For example, the processor (110) can generate a prompt by inserting keywords into placeholders of a prompt template.

[0066] For example, the processor (110) may generate a prompt to request an expected image of the completed food by inputting a first image, a type of food, and operation information into a predefined prompt format. For example, a rule for the processor (110) to generate a prompt based on the first image, a type of food, and operation information may be stored in memory (130).

[0067] FIG. 5 is a diagram illustrating the operation of a cooking device generating a prompt according to one embodiment.

[0068] Referring to FIG. 5, the user can input operation information (520) using an input interface (160) (e.g., a touch screen) included in the display (510).

[0069] According to one embodiment, the processor (110) can generate a prompt by inserting a first image, a type of food, and operation information into a prompt template such as “a photo of Delicious-looking {type of food and first image} cooked in {cooking mode} mode set at {cooking time} minutes and {cooking temperature} degrees”.

[0070] For example, let us assume that a user inputs a cooking time (e.g., 1 hour), a cooking temperature (e.g., 160 degrees), and a cooking mode (convection) through an input interface (160), and that the type of food is pizza. The processor (110) can generate a prompt (530) including “a photo of Delicious-looking pizza cooked in convection mode set at 60 minutes and 160 degrees” and a first image.

[0071] According to one embodiment, in operation 450, the processor (110) may acquire an image (hereinafter referred to as the second image) based on a prompt. The second image may include an expected image of the cooked food when the cooking of the food included in the first image is completed.

[0072] For example, the processor (110) can generate a second image by inputting a prompt to an image generation model (hereinafter referred to as the first artificial intelligence model). The first artificial intelligence model may be a model trained using multiple images of food before cooking, multiple images of food after cooking is completed, and information on the type and operation of food as input data, and an image of food expected to be cooked as output data. For example, the first artificial intelligence model may be a generative adversarial network (GAN) or a Diffusion-based generative model using a Transformer structure.

[0073] The fact that an artificial intelligence model is trained means that a basic artificial intelligence model (e.g., an artificial intelligence model containing arbitrary random parameters) is trained using a number of training data by a learning algorithm, thereby creating a predefined behavioral rule or artificial intelligence model set to perform a desired characteristic (or purpose). This training may be performed through a separate server and / or system, but is not limited thereto, and may also be performed in the cooking device (100). Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples mentioned above.

[0074] According to one embodiment, the first artificial intelligence model may be stored in memory (130), but is not necessarily limited thereto. For example, the first artificial intelligence model may be stored on an external server, and the processor (110) may receive information about the expected cooking completion image after sending a prompt to the external server.

[0075] According to one embodiment, in operation 460, the processor (110) can obtain an image (hereinafter referred to as the third image) by photographing the cooking chamber (10) through the camera (120). The processor (110) can obtain the third image using the camera (120) at preset time intervals from the time the cooking device (100) starts operating. The preset time may be set at manufacturing or initial use, or may be set and changed by user input. For example, information regarding the preset time may be stored in the memory (130).

[0076] For example, the processor (110) can acquire a third image at 1-minute intervals from the time the cooking device (100) starts operating until the time it ends operating.

[0077] According to one embodiment, in operation 470, the processor (110) can control the operation of the cooking device (100) based on the similarity between the second image and the third image. The similarity between the second image and the third image may mean a numerical value representing the degree of similarity between the second image and the third image.

[0078] For example, the processor (110) can obtain the similarity between the second image and the third image based on the cosine similarity between the feature vector of the second image and the feature vector of the third image. The processor (110) can obtain the feature vectors of the second image and the third image using a CNN. The processor (110) can obtain the similarity between the feature vector of the second image and the feature vector of the third image using a cosine similarity formula. In this case, the closer the cosine similarity between the second image and the third image is to 1, the more similar the two images are.

[0079] However, the method for determining the similarity between the second image and the third image is not limited to the examples described above. For instance, the method for determining the similarity between the second image and the third image may include, but is not limited to, pixel-based methods (e.g., MSE (mean squared error), PSNR (peak signal-to-noise ratio) or structural similarity determination methods (e.g., SSIM (structural similarity index)).

[0080] The processor (110) can control the heating unit (180) of the cooking device (100) based on the similarity between the second image and the third image. For example, the processor (110) can change the operation information of the cooking device (100) based on the similarity. A detailed description of how the cooking device (100) controls the operation of the cooking device (100) based on the similarity is described in FIGS. 6 to 8.

[0081] FIG. 6 is a flowchart illustrating the operation of a cooking device when the similarity is greater than or equal to a preset value, according to one embodiment.

[0082] According to one embodiment, in operation 610, the processor (110) can identify the similarity between the second image and the third image. According to one embodiment, in operation 620, the processor (110) can identify whether the similarity is greater than or equal to a preset value (hereinafter referred to as the first value). For example, the first value may be set at the time of manufacturing or initial use of the cooking device (100), or may be set and changed by user input. For example, information regarding the first value may be stored in memory (130).

[0083] According to one embodiment, in operation 620-Y and operation 630, the processor (110) may stop the operation of the cooking device (100) if it is identified that the similarity is greater than or equal to a first value. Stopping the operation of the cooking device (100) may mean stopping the transfer of heat to the food by turning off the heating unit (180). For example, the processor (110) may change the remaining operating time of the cooking device (100) to 0.

[0084] The fact that the similarity is identified as being greater than or equal to the first value is equivalent to the fact that the cooking state of the food at the time the third image was acquired is identified as being similar to the cooking state of the food at the time the cooking is completed. Therefore, the cooking device (100) can provide the user with food that is cooked without being overcooked by terminating the operation of the cooking device (100) without separate user intervention.

[0085] FIG. 7 is a flowchart illustrating the operation of a cooking device when the similarity is greater than or equal to a preset value, according to one embodiment.

[0086] According to one embodiment, in operation 710, the processor (110) can identify the similarity between the second image and the third image. According to one embodiment, in operation 720, the processor (110) can identify whether the similarity is greater than or equal to a preset value (hereinafter referred to as the second value). For example, the second value may be set during the manufacture or initial use of the cooking device (100), or may be set and changed by user input. For example, information regarding the second value may be stored in memory (130). The second value may be the same value as the aforementioned first value or may be different values.

[0087] According to one embodiment, in operation 720-Y and operation 730, if the processor (110) identifies that the similarity is greater than or equal to a second value, the processor (110) may reduce the cooking time set by user input based on the similarity.

[0088] For example, the higher the similarity between the second and third images, the greater the reduction in cooking time can be. In this case, the reduced cooking time may be smaller than the remaining cooking time. The remaining cooking time may be the difference between the cooking time and the time corresponding to when the third image was acquired.

[0089] For example, the degree of reduction in cooking time based on similarity may be set during manufacturing or initial use, or may be set and changed by user input. The degree of reduction in cooking time based on similarity may be stored in memory (130) in the form of a lookup table. For example, let us assume that the remaining cooking time is 30 minutes and the second value is 0.8. The processor (110) may reduce the cooking time by 20 minutes if the similarity between the second image and the third image is 0.95, and reduce the cooking time by 15 minutes if the similarity is 0.85.

[0090] For example, the processor (110) may divide the similarity between the second image and the third image into at least one range and determine the degree of reduction in cooking time corresponding to each range. The range of similarity and the corresponding degree of reduction in cooking time may be stored in memory (130). For example, when the second value is 0.8, the processor (110) may reduce the remaining cooking time by 50% if the similarity is 0.8 or higher and less than 0.9, and reduce the remaining cooking time by 90% if the similarity is 0.9 or higher.

[0091] However, it is not limited to this, and the processor (110) may determine the remaining cooking time based on similarity, rather than obtaining the degree of reduction of cooking time based on similarity. The remaining cooking time based on similarity may be stored in memory (130). For example, the processor (110) may determine the remaining cooking time as 5 minutes when the second value is 0.8 and the similarity is 0.95. For example, the processor (110) may determine 5% (e.g., 5 minutes) of the cooking time (e.g., 100 minutes) included in the operation information as the remaining cooking time when the second value is 0.8 and the similarity is 0.95.

[0092] According to one embodiment, if the similarity is identified as being greater than or equal to a second value, the processor (110) may lower the cooking temperature set by user input based on the similarity. For example, the degree of reduction in cooking temperature according to similarity may be set during manufacturing or initial use, or may be set and changed by user input. For example, the higher the similarity between the second image and the third image, the more the cooking temperature may be lowered.

[0093] For example, the degree of cooking temperature reduction according to similarity may be stored in memory (130) in the form of a lookup table. For example, the processor (110) may divide the similarity of the second image and the third image into at least one range and determine the degree of cooking temperature reduction corresponding to each range.

[0094] The processor (110) can change operation information based on the reduced cooking time or cooking temperature. The fact that the similarity is identified as being greater than or equal to the second value is equivalent to the fact that the cooking state of the food at the time the third image was acquired is identified as being similar to the cooking state of the food after cooking is complete. Therefore, the cooking device (100) can control the food to be properly cooked by reducing the cooking temperature or cooking time of the cooking device (100) without any intervention by a separate user.

[0095] FIG. 8 is a flowchart illustrating the operation of a cooking device when the similarity is smaller than a preset value, according to one embodiment.

[0096] According to one embodiment, in operation 810, the processor (110) may acquire a third image at a time corresponding to the end time of the operation of the cooking device (100). The time corresponding to the end time of the operation of the cooking device (100) may mean the time closest to the end time of the operation of the cooking device (100) among a plurality of times when the cooking device (100) acquires the third image using the camera (120). However, it is not limited thereto, and the time corresponding to the end time of the operation may mean the time when the third image is acquired after a preset time during the time when the cooking device (100) is operating.

[0097] According to one embodiment, in operation 820, the similarity between the second image and the third image can be identified. In operation 830, the processor (110) can identify whether the similarity is smaller than a preset value (hereinafter referred to as the third value). The third value may be the same value as the aforementioned first value or second value, or may be a different value. For example, the third value may be set during the manufacture or initial use of the cooking device (100), or may be set and changed by user input. For example, information regarding the third value may be stored in memory (130).

[0098] According to one embodiment, in operation 830-Y and operation 840, if the processor (110) identifies that the similarity is smaller than a third value, it may increase the cooking time based on the similarity.

[0099] For example, the lower the similarity between the second and third images, the greater the cooking time can be increased. For instance, the increase in cooking time based on similarity may be set during manufacturing or initial use, or it may be set and changed by user input.

[0100] For example, let us assume that the third value is 0.8 and the remaining cooking time is 5 minutes. The processor (110) can increase the cooking time by 20 minutes if the similarity between the second image and the third image is 0.75, and increase the cooking time by 30 minutes if the similarity is 0.7.

[0101] For example, the processor (110) may divide the similarity between the second image and the third image into at least one range and determine the degree of increase in cooking time corresponding to each range. The range of similarity and the corresponding degree of decrease in cooking time may be stored in memory (130). For example, the processor (110) may increase the cooking time by 10 minutes when the similarity is 0.7 or higher and less than 0.8, and increase the remaining cooking time by 20 minutes when the similarity is less than 0.7.

[0102] However, it is not limited to this, and the processor (110) can determine the remaining cooking time based on similarity, rather than obtaining the degree of increase in cooking time based on similarity. The remaining cooking time based on similarity may be stored in memory (130). For example, the processor (110) can determine the remaining cooking time as 10 minutes when the third value is 0.8 and the similarity is 0.75. For example, the processor (110) can determine 20% (e.g., 20 minutes) of the cooking time (e.g., 100 minutes) as the remaining cooking time when the second value is 0.8 and the similarity is 0.65.

[0103] According to one embodiment, if the processor (110) identifies that the similarity is smaller than a third value, it can increase the cooking temperature set by user input based on the similarity.

[0104] For example, the degree of increase in cooking temperature based on similarity may be set during manufacturing or initial use, or may be set and changed by user input. For example, the lower the similarity between the second image and the third image, the greater the cooking temperature can be increased.

[0105] For example, the processor (110) can divide the similarity between the second image and the third image into at least one range and determine the degree of increase of the cooking temperature corresponding to each range.

[0106] The processor (110) can change operation information based on the increased cooking time or cooking temperature. If the similarity is identified as being smaller than the third value, it may mean that the food at the time the third image was acquired is not cooked properly. Therefore, the cooking device (100) can provide the user with food cooked to be properly cooked by increasing the cooking temperature or cooking time of the cooking device (100) without separate user intervention.

[0107] FIG. 9 is a drawing illustrating a display screen according to the operation of a cooking device according to one embodiment.

[0108] According to one embodiment, the cooking device (100) may display a UI (910) on a display to guide a user in placing food into the cooking chamber (10). When the user places food into the cooking chamber (10), the cooking device (100) may use a camera (120) to photograph the cooking chamber (10) and obtain a first image.

[0109] The cooking device (100) can identify the type of food located in the cooking room (10) by inputting a first image into a second artificial intelligence model. If there are multiple types of food identified, the cooking device (100) can display a UI (920) on the display to guide the user to select one of the multiple types of food. The cooking device (100) can identify the type of food based on user input detected through the input interface (160). For example, if multiple types of food are identified, the cooking device (100) can display a UI (920) (e.g., cheese gratin and pizza) corresponding to the identified food on the display. If the cooking device (100) detects a user touch input corresponding to potato pizza among cheese gratin or pizza, the cooking device (100) can identify the type of food as pizza.

[0110] According to one embodiment, the cooking device (100) may display a UI (920) on a display that allows selecting a type of food among a plurality of food types based on the identification of a plurality of food types based on a first image. The cooking device (100) may identify a type of food corresponding to the user input among a plurality of food types based on user input to the UI (920). The cooking device (100) may display a UI (940) including the identified type of food on a display.

[0111] According to one embodiment, the cooking device (100) may display operation information (930) of the cooking device (100) determined based on user input on a display. For example, if the operation information entered by the user is convection mode, 350 degrees Fahrenheit, and 1 hour and 30 minutes, the cooking device (100) may display information (930) on a display indicating that the cooking device (100) operates based on convection mode, 350 degrees Fahrenheit, and 1 hour and 30 minutes.

[0112] According to one embodiment, the cooking device (100) can display (940) a second image (e.g., an image of expected cooking completion) obtained based on a first image, a type of food, and operation information.

[0113] According to one embodiment, the cooking device (100) can control the operation of the cooking device (100) based on the similarity between the third image and the second image obtained by photographing the cooking room (10) through a camera (120).

[0114] FIG. 10 is a flowchart illustrating a method for controlling operation information of a cooking device based on a thermal image of a cooking chamber taken according to one embodiment.

[0115] According to one embodiment, the cooking device (100) may include a thermal imaging camera. The thermal imaging camera may acquire a thermal image. Specifically, the thermal imaging camera may detect radiant heat emitted by an object and acquire a thermal image showing the intensity and distribution of the detected heat.

[0116] When a user input to operate the cooking device (100) is received, the cooking device (100) can irradiate microwaves into the cooking chamber (10). Then, when the temperature of the thermal imaging camera rises due to the microwaves, the cooking device (100) can take a picture of the cooking chamber (10) using the thermal imaging camera to obtain a thermal image.

[0117] And, the cooking device (100) can identify the rate of change of temperature of at least one of the background area and the turntable area based on the acquired thermal image. The background area may include the internal space excluding the turntable in the cooking chamber (10).

[0118] According to one embodiment, the cooking device (100) can acquire a thermal image (1010) (hereinafter referred to as the first thermal image) using a thermal imaging camera while operating the cooking device (100) based on user input. According to one embodiment, the cooking device (100) can acquire a thermal image (1020) (hereinafter referred to as the second thermal image) using a thermal imaging camera at a point in time after a certain period has elapsed since operating the cooking device (100).

[0119] According to one embodiment, a cooking device (100) may generate a prompt to request an expected thermal image of the finished food based on a first thermal image (1010), a second thermal image (1020), and the type of food. The cooking device (100) may input the prompt into an artificial intelligence model to obtain an expected thermal image (1030) of the finished food (hereinafter referred to as the third thermal image). The artificial intelligence model may be a model trained using the first thermal image (1010), the second thermal image (1020), and the type of food as input data, and the expected thermal image of the finished cooking as output data.

[0120] According to one embodiment, the cooking device (100) can obtain a thermal image (hereinafter referred to as the fourth thermal image) by photographing the cooking room (10) through a thermal imaging camera. The processor (110) can obtain the fourth thermal image using the thermal imaging camera at preset time intervals starting from the time the cooking device (100) begins operation.

[0121] According to one embodiment, the cooking device (100) can control the operation of the cooking device (100) based on the similarity between the third thermal image and the fourth thermal image. The description of controlling the operation of the cooking device (100) based on similarity is described in detail in FIGS. 6 to 8, so redundant content is omitted.

[0122] According to one embodiment of the present disclosure, a cooking device (100) including a cooking chamber (10) may include at least one processor including a memory (130) for storing camera (120) instructions for photographing the cooking chamber (10) and a processing circuitry.

[0123] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device (100) may heat food inside the cooking chamber (10) in response to operation information set by user input, and the cooking device (100) may obtain a first image by photographing the cooking chamber (10) through the camera (120), identify the type of food based on the first image, generate a prompt to request an expected image of the time when the cooking of the food is completed based on the first image, the type of food, and the operation information, input the prompt into a first artificial intelligence model to obtain a second image, obtain a third image by photographing the cooking chamber (10) through the camera (120), and control the operation of the cooking device (100) based on the similarity between the second image and the third image.

[0124] For example, the above operation information may include at least one of a cooking time, a cooking temperature, a cooking completion time, and a cooking mode.

[0125] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device (100) may be stopped based on the fact that the similarity between the second image and the third image is greater than or equal to a preset value.

[0126] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device (100) may reduce the cooking time set by the user input or lower the cooking temperature set by the user input based on the similarity between the second image and the third image being greater than or equal to a preset value.

[0127] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device (100) may acquire the third image using the camera (120) at preset time intervals from the time the cooking device (100) starts operation.

[0128] For example, when the above instructions are executed individually or collectively by the at least one processor, if the cooking device (100) identifies that the similarity between the second image and the third image obtained at a time corresponding to the end of the operation is smaller than a preset value, it may increase the cooking time set by the user input or increase the cooking temperature set by the user input based on the similarity.

[0129] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device (100) may input the first image into the second artificial intelligence model to identify the type of food.

[0130] For example, the cooking device (100) further includes a display, and when the instructions are executed individually or collectively by the at least one processor, the cooking device (100) may display the second image on the display.

[0131] For example, the cooking device (100) further includes a display, and when the instructions are executed individually or collectively by the at least one processor, the cooking device displays a UI on the display that allows the user to select a type of food among the plurality of food types based on the identification of the plurality of food types based on the first image, and can determine the type of food based on user input selecting a type of food among the plurality of food types through the UI.

[0132] According to one embodiment of the present disclosure, a method for controlling a cooking device (100) including a cooking chamber (10) may be provided.

[0133] The above control method may include the steps of: obtaining a first image by photographing the cooking chamber (10) through a camera (120) included in the cooking device (100) based on heating food inside the cooking chamber (10) in response to operation information set by user input; identifying the type of food based on the first image; generating a prompt to request an expected image of the time when the cooking of the food is completed based on the first image, the type of food, and the operation information; obtaining a second image by inputting the prompt into a first artificial intelligence model; obtaining a third image by photographing the cooking chamber (10) through the camera (120); and controlling the operation of the cooking device (100) based on the similarity between the second image and the third image.

[0134] For example, the above operation information may include at least one of a cooking time, a cooking temperature, a cooking completion time, and a cooking mode.

[0135] For example, the step of controlling the operation of the cooking device (100) may include the step of stopping the operation of the cooking device (100) based on the fact that the similarity between the second image and the third image is greater than or equal to a preset value.

[0136] For example, the step of controlling the operation of the cooking device may further include the step of performing at least one of shortening the cooking time set by the user input or lowering the cooking temperature set by the user input, or stopping the operation of the cooking device, based on the fact that the similarity between the second image and the third image is greater than or equal to a preset value.

[0137] For example, the step of controlling the operation of the cooking device (100) may include reducing the cooking time set by the user input or lowering the cooking temperature set by the user input based on the similarity between the second image and the third image being greater than or equal to a preset value.

[0138] For example, the step of acquiring the third image may include acquiring the third image using the camera (120) at preset time intervals from the time when the cooking device (100) starts operating.

[0139] For example, the step of controlling the operation of the cooking device (100) may include, if it is identified that the similarity between the second image and the third image obtained at a time corresponding to the end of the operation is smaller than a preset value, increasing the cooking time set by the user input or increasing the cooking temperature set by the user input based on the similarity.

[0140] For example, the step of controlling the operation of the cooking device may further include the step of increasing the cooking time set by the user input or increasing the cooking temperature set by the user input based on the fact that the similarity between the second image and the third image acquired at a time corresponding to the end time of the operation is identified as being smaller than a preset value.

[0141] For example, the step of identifying the type of food may include the step of inputting the first image into a second artificial intelligence model to identify the type of food.

[0142] For example, the method may further include the step of displaying the second image on a display included in the cooking device (100).

[0143] For example, the control method may further include the step of displaying a UI on a display that allows selecting a type of food among a plurality of food types based on the identification of a plurality of food types based on the first image, and the step of identifying a type of food corresponding to the user input among the plurality of food types as the type of food based on user input to the UI.

[0144] For example, the control method may further include the step of displaying a UI on a display included in the cooking device to select a type of food among the plurality of food types based on the identification of a plurality of food types based on the first image, and the step of determining the type of food based on user input selecting a type of food among the plurality of food types through the UI.

[0145] Although various embodiments have been described above, each embodiment is not necessarily implemented individually, and may be combined with at least one other embodiment, either wholly or partially, to be implemented together in a single product.

[0146] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include other data of modulated data signals, such as computer-readable instructions, data structures, or program modules.

[0147] Additionally, computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage media' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, 'non-transitory storage media' may include a buffer in which data is stored temporarily.

[0148] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0149] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0150] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.

Claims

1. In a cooking device including a cooking chamber, A camera for photographing the above-mentioned kitchen; Memory for storing instructions; and At least one processor including processing circuitry; comprising, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, Based on cooking food inside the cooking chamber in response to operation information set by user input, the cooking chamber is photographed through the camera to obtain a first image, and Identify the type of food based on the first image above, and Generating a prompt to request an expected image of the time when the cooking of the food is completed based on at least one of the first image and the type of the food and the operation information, and The above prompt is input into the first model to obtain a second image, and A third image is obtained by photographing the kitchen through the camera above, and A cooking device that controls the operation of the cooking device based on the second image and the third image.

2. In Paragraph 1, The above operation information is, A cooking device comprising at least one of a cooking time, a cooking temperature, a cooking completion time, and a cooking mode.

3. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, A cooking device that, based on the similarity between the second image and the third image being greater than or equal to a preset value, performs at least one of shortening the cooking time set by the user input or lowering the cooking temperature set by the user input, or stops the operation of the cooking device.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, A cooking device that acquires the third image using the camera at preset time intervals during the cooking operation of the cooking device.

5. In Paragraph 4, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, A cooking device that increases the cooking time set by the user input or increases the cooking temperature set by the user input based on the fact that the similarity between the second image and the third image obtained at a time corresponding to the end time of the operation is identified as being smaller than a preset value.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, A cooking device that inputs the first image above into a second model to identify the type of food above.

7. In Paragraph 1, Including a display; further When the above instructions are executed individually or collectively by the at least one processor, the cooking device, A cooking device that displays the above second image on the display.

8. In Paragraph 1, Including a display; further When the above instructions are executed individually or collectively by the at least one processor, the cooking device, A UI is displayed on the display to allow selecting a type of food among the plurality of food types based on the identification of a plurality of food types based on the first image above, and A cooking device that determines the type of food based on user input selecting the type of food among the plurality of food types through the above UI.

9. A method for controlling a cooking device including a cooking chamber, A step of obtaining a first image by photographing the cooking chamber through a camera included in the cooking device, based on heating food inside the cooking chamber in response to operation information set by user input; A step of identifying the type of food based on the first image above; A step of generating a prompt to request an expected image of the time when the cooking of the food is completed based on at least one of the first image and the type of food and the operation information; A step of acquiring a second image by inputting the above prompt into a first model; A step of obtaining a third image by photographing the kitchen through the camera; and A control method comprising the step of controlling the operation of the cooking device based on the second image and the third image.

10. In Paragraph 9, The above operation information is, A control method comprising at least one of a cooking time, a cooking temperature, a cooking completion time, and a cooking mode.

11. In Paragraph 9, The step of controlling the operation of the above cooking device is A control method comprising the step of, based on the fact that the similarity between the second image and the third image is greater than or equal to a preset value, performing at least one of shortening the cooking time set by the user input or lowering the cooking temperature set by the user input, or stopping the operation of the cooking device.

12. In Paragraph 9, The step of acquiring the third image above is, A control method comprising the step of acquiring the third image using the camera at preset time intervals from the time the cooking device starts operating.

13. In Paragraph 12, The step of controlling the operation of the above cooking device is, A control method comprising the step of increasing the cooking time set by the user input or increasing the cooking temperature set by the user input based on the fact that the similarity between the second image and the third image obtained at a time corresponding to the end time of the operation is identified as being smaller than a preset value.

14. In Paragraph 9, The step of identifying the type of food mentioned above is, A control method comprising the step of inputting the first image into a second model to identify the type of food.

15. In Paragraph 8, A step of displaying a UI on a display included in the cooking device to select a type of food among the plurality of food types based on the identification of a plurality of food types based on the first image above; A control method further comprising the step of determining the type of food based on user input selecting the type of food among the plurality of food types through the above UI.

Images