Cooking apparatus and control method thereof

Abstract

A cooking apparatus including a cooking chamber is disclosed. The cooking apparatus comprises: a camera for photographing the cooking chamber; a communication circuit; a memory; and at least one processor which: identifies at least one food ingredient in the cooking chamber on the basis of a first image acquired through the camera; if a first food ingredient is selected from among the at least one food ingredient on the basis of a user input, acquires a plurality of predicted images corresponding to a cooking state of the first food ingredient predicted according to a cooking operation of the cooking apparatus; receives a user input for selecting a predicted image from among the plurality of predicted images; acquires a second image by photographing the cooking chamber by using the camera while the cooking apparatus performs the cooking operation; and if it is identified that a cooking state of the first food ingredient in the second image corresponds to the predicted image, transmits a notification of the first food ingredient to an external electronic device through the communication circuit.

Description

Cooking device and control method thereof

[0001] The present disclosure relates to a cooking device and a method for controlling the same, and more specifically, to a cooking device and a method for controlling the same that control the cooking environment based on a captured image of a food ingredient.

[0002] A cooking device is a device configured to cook food. For example, a cooking device may include an oven, induction cooktop, electric range, gas range, etc. A service is being developed that provides an image of the cooking result using a camera equipped on the cooking device to capture images of the ingredients to be cooked.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] According to one embodiment of the present disclosure, a cooking device including a cooking chamber comprises at least one processor including a camera for photographing the cooking chamber, a communication circuit, a memory for storing instructions, and a processing circuitry. When the instructions are executed individually or collectively by the at least one processor, the cooking device identifies at least one food ingredient in the cooking chamber based on a first image acquired through the camera, and when the first food ingredient is selected among the at least one food ingredient based on user input, acquires a plurality of predicted images corresponding to the cooking state of the first food ingredient expected according to the cooking operation of the cooking device, receives user input for selecting a predicted image among the plurality of predicted images, and while the cooking device performs the cooking operation, photographs the cooking chamber using the camera to acquire a second image, and when the cooking state of the first food ingredient in the second image is identified as corresponding to the predicted image, transmits a notification regarding the first food ingredient to an external electronic device through the communication circuit.

[0005] According to one embodiment of the present disclosure, a method for controlling a cooking device including a cooking chamber comprises: identifying at least one food ingredient in the cooking chamber based on a first image acquired through a camera included in the cooking device; when the first food ingredient is selected among the at least one food ingredient based on user input, acquiring a plurality of predicted images corresponding to the cooking state of the first food ingredient expected according to the cooking operation of the cooking device; receiving user input for selecting a predicted image among the plurality of predicted images; capturing the cooking chamber using the camera while the cooking device performs a cooking operation to acquire a second image; and when the cooking state of the first food ingredient in the second image is identified as corresponding to the predicted image, transmitting a notification regarding the first food ingredient to an external electronic device.

[0006] According to one embodiment of the present disclosure, a non-transient computer-readable recording medium stores one or more instructions executed by a processor of a cooking device to perform an operation, wherein the operation comprises: identifying at least one food ingredient in the cooking chamber based on a first image acquired through a camera included in the cooking device; when the first food ingredient is selected among the at least one food ingredient based on user input, acquiring a plurality of predicted images corresponding to the cooking state of the first food ingredient expected according to the cooking operation of the cooking device; receiving user input for selecting a predicted image among the plurality of predicted images; capturing the cooking chamber using the camera while the cooking device performs the cooking operation to acquire a second image; and when the cooking state of the first food ingredient in the second image is identified as corresponding to the predicted image, transmitting a notification regarding the first food ingredient to an external electronic device.

[0007] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0008] FIG. 1 is a drawing for illustrating an exemplary home appliance capable of performing the operations described in this document according to one embodiment.

[0009] FIG. 2 is a set of drawings for explaining the structure and operation of a cooking device and an external electronic device to aid in understanding the present disclosure, according to one embodiment.

[0010] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0011] FIG. 4 is a block diagram illustrating the configuration of a cooking device according to one embodiment.

[0012] FIG. 5 is a block diagram illustrating the detailed configuration of a cooking device according to one embodiment.

[0013] FIG. 6 is a flowchart illustrating an example of a method in which a cooking device provides a notification about food ingredients inside a cooking chamber according to one embodiment.

[0014] FIG. 7 is a drawing illustrating an example of an operation in which a cooking device (100) identifies food ingredients based on a first image according to one embodiment.

[0015] FIG. 8 is a diagram illustrating an example of an operation in which a cooking device (100) identifies a first food ingredient among at least one food ingredient based on user input, according to one embodiment.

[0016] FIG. 9 is a diagram illustrating an example of an operation in which a cooking device (100) selects a predicted image from a plurality of predicted images based on user input, according to one embodiment.

[0017] FIG. 10 is a drawing illustrating an example in which a cooking device, according to one embodiment, provides a notification of a first food ingredient to an external electronic device.

[0018] FIG. 11 is a drawing illustrating an example of a method in which a cooking device provides notifications about a plurality of food ingredients inside a cooking chamber, according to one embodiment.

[0019] FIG. 12 is a diagram illustrating an example of a method in which a cooking device provides a notification about all food ingredients inside a cooking chamber according to one embodiment.

[0020] FIG. 13 is a diagram illustrating an example of a method in which a cooking device provides a notification regarding a food ingredient when the food ingredient is not recognized, according to one embodiment.

[0021] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments.

[0022] In relation to the description of the drawings, similar reference numerals may be used for similar or related components.

[0023] The singular form of the noun corresponding to an item may include one or plural items, unless the relevant context clearly indicates otherwise.

[0024] In this document, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0025] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0026] Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that the component may be connected to the other component directly (e.g., via a wire), wirelessly, or through a third component.

[0027] Terms such as “include” or “have” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this document, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0028] When it is said that one component is “connected,” “combined,” “supported,” or “in contact” with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0029] When it is said that a component is located “on” another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0030] The term “and / or” includes a combination of multiple related described components or any of the multiple related described components.

[0031] The operating principle and embodiments of the present invention will be described below with reference to the attached drawings.

[0032] FIG. 1 is a drawing for illustrating an exemplary home appliance capable of performing the operations described in this document according to one embodiment.

[0033] The home appliance (10) may include a communication module capable of communicating with other home appliances, a user device (2) or a server (3), a user interface that receives user input or outputs information to the user, at least one processor that controls the operation of the home appliance (10), and at least one memory in which a program for controlling the operation of the home appliance (10) is stored.

[0034] The home appliance (10) may be at least one of various types of home appliances. For example, the home appliance (10) may include at least one of a refrigerator (11), a dishwasher (12), an electric range (13), an electric oven (14), an air conditioner (15), a garment care device (16), a washing machine (17), a dryer (18), and a microwave oven (19) as illustrated, but is not limited thereto, and may include various types of home appliances such as a cleaning robot, a vacuum cleaner, and a television not illustrated in the drawings. In addition, the aforementioned home appliances are merely examples, and in addition to the aforementioned home appliances, a device capable of performing the operation described below by connecting to other home appliances, a user device (2), or a server (3) may be included in the home appliance (10) according to one embodiment.

[0035] The server (3) may include a communication module capable of communicating with another server, a home appliance (10), or a user device (2), at least one processor capable of processing data received from another server, a home appliance (10), or a user device (2), and at least one memory capable of storing a program for processing data or processed data. This server (3) may be implemented as various computing devices such as a workstation, a cloud, a data drive, or a data station. The server (3) may be implemented as one or more servers physically or logically separated based on functions, detailed configurations of functions, or data, and may transmit and receive data and process the transmitted and received data through communication between each server.

[0036] The server (3) can perform functions such as managing user accounts, registering home appliances (10) associated with user accounts, and managing or controlling the registered home appliances (10). For example, a user can create a user account by accessing the server (3) through a user device (2). A user account can be identified by an ID and password set by the user. The server (3) can register home appliances (10) to the user account according to a set procedure. For example, the server (3) can register, manage, and control home appliances (10) by linking identification information of the home appliance (10) (e.g., serial number or MAC address, etc.) to the user account. The user device (2) may include a communication module capable of communicating with the home appliance (10) or the server (3), a user interface that receives user input or outputs information to the user, at least one processor that controls the operation of the user device (2), and at least one memory in which a program for controlling the operation of the user device (2) is stored.

[0037] The user device (2) may be carried by the user or placed in the user's home or office, etc. The user device (2) may include, but is not limited to, a personal computer, terminal, portable telephone, smartphone, handheld device, wearable device, etc.

[0038] A program, i.e., an application, for controlling the home appliance (10) can be stored in the memory of the user device (2). The application may be sold with the user device (2) already installed, or it may be downloaded and installed from an external server.

[0039] By running an application installed on the user device (2), the user can connect to the server (3) to create a user account, and communicate with the server (3) based on the logged-in user account to register the home appliance device (10).

[0040] For example, if the home appliance (10) is operated in accordance with the procedure guided by the application installed on the user device (2) so that the home appliance (10) can be connected to the server (3), the home appliance (10) can be registered to the user account by registering the identification information of the home appliance (10) (e.g., serial number or MAC address, etc.) to the user account on the server (3).

[0041] The user can control the home appliance (10) using an application installed on the user device (2). For example, when the user logs into the user account using an application installed on the user device (2), the home appliance (10) registered to the user account appears, and when the user inputs a control command for the home appliance (10), the control command can be transmitted to the home appliance (10) through the server (3).

[0042] A network may include both wired and wireless networks. Wired networks include cable networks or telephone networks, etc., and wireless networks may include all networks that transmit and receive signals via radio waves. Wired and wireless networks may be connected to each other.

[0043] Networks may include wide area networks (WANs) such as the Internet, local area networks (LANs) formed around access points (APs), and short-range wireless networks that do not pass through access points (APs). Short-range wireless networks may include, but are not limited to, Bluetooth (IEEE 802.15.1), Zigbee (IEEE 802.15.4), Wi-Fi Direct, Near Field Communication (NFC), Z-Wave, etc.

[0044] The access point (AP) can connect a home appliance (10) or a user device (2) to a wide area network (WAN) to which a server (3) is connected. The home appliance (10) or the user device (2) can be connected to the server (3) through the wide area network (WAN).

[0045] The access point (AP) can communicate with a home appliance (10) or user device (2) using wireless communication such as Wi-Fi (Wi-Fi™, IEEE 802.11), Bluetooth (Bluetooth™, IEEE 802.15.1), and Zigbee (Zigbee, IEEE 802.15.4), and can connect to a wide area network (WAN) using wired communication, but is not limited thereto.

[0046] According to various embodiments, the home appliance (10) may be directly connected to the user device (2) or server (3) without going through the access relay (AP).

[0047] The home appliance (10) can be connected to a user device (2) or server (3) via a long-distance wireless network or a short-distance wireless network.

[0048] For example, the home appliance (10) can be connected to the user device (2) via a short-range wireless network (e.g., Wi-Fi Direct).

[0049] As another example, the home appliance (10) can be connected to a user device (2) or server (3) via a wide area network (WAN) using a long-distance wireless network (e.g., a cellular communication module).

[0050] As another example, a home appliance (10) can be connected to a wide area network (WAN) using wired communication and can be connected to a user device (2) or a server (3) through the wide area network (WAN).

[0051] If the home appliance (10) can connect to a wide area network (WAN) using wired communication, it may operate as a connection relay. Accordingly, the home appliance (10) can connect other home appliances to the wide area network (WAN) to which the server (3) is connected. Additionally, other home appliances can connect the home appliance (10) to the wide area network (WAN) to which the server (3) is connected.

[0052] The home appliance (10) can transmit information regarding operation or status to other home appliances, user devices (2), or servers (3) via a network. For example, the home appliance (10) can transmit information regarding operation or status to other home appliances, user devices (2), or servers (3) when a request is received from the server (3), when a specific event occurs in the home appliance (10), or periodically or in real time. When the server (3) receives information regarding operation or status from the home appliance (10), it can update the stored information regarding operation or status of the home appliance (10) and transmit the updated information regarding operation and status of the home appliance (10) to the user devices (2) via a network. Here, updating information may include various operations that change existing information, such as adding new information to existing information or replacing existing information with new information.

[0053] The home appliance (10) can obtain various information from other home appliances, user devices (2), or servers (3) and provide the obtained information to the user. For example, the home appliance (10) can obtain information related to the functions of the home appliance (10) (e.g., recipes, laundry methods, etc.) and various environmental information (e.g., weather, temperature, humidity, etc.) from the server (3), and can output the obtained information through a user interface.

[0054] The home appliance (10) may operate according to control commands received from other home appliances, user devices (2), or servers (3). For example, if the home appliance (10) has obtained prior approval from a user to operate according to control commands from servers (3) even without user input, the home appliance (10) may operate according to control commands received from servers (3). Here, the control commands received from servers (3) may include, but are not limited to, control commands entered by the user through user devices (2) or control commands based on pre-set conditions.

[0055] The user device (2) can transmit information about the user to the home appliance (10) or server (3) through a communication module. For example, the user device (2) can transmit information about the user's location, health status, preferences, schedule, etc. to the server (3). The user device (2) can transmit information about the user to the server (3) upon the user's prior approval.

[0056] The home appliance (10), user device (2), or server (3) may determine control commands using technology such as artificial intelligence. For example, the server (3) may receive information regarding the operation or status of the home appliance (10) or information regarding the user of the user device (2), process it using technology such as artificial intelligence, and transmit the processing result or control command to the home appliance (10) or user device (2) based on the processing result.

[0057] FIG. 2 is a set of drawings for explaining the structure and operation of a cooking device and an external electronic device to aid in understanding the present disclosure, according to one embodiment.

[0058] According to one embodiment, the cooking device (100) may include a cooking chamber (20) and a camera (130). The cooking device (100) is a device that performs various cooking operations by applying high-temperature heat generated using electricity or gas to food ingredients, and may be an oven, induction cooker, electric range, or gas range, but is not limited to a specific type.

[0059] According to one embodiment, the cooking device (100) can cook food based on user input related to the operation of the cooking device. The user can place food to be cooked into the cooking chamber (20) and input operation information (e.g., cooking mode, operating temperature, operating time). The cooking device (100) can cook food according to the operating temperature, operating time, and cooking mode corresponding to the operation information.

[0060] The operating temperature may refer to the temperature inside the cooking device (100). The cooking device (100) may maintain the internal temperature at the operating temperature using a heater mounted on the cooking device. The operating time may refer to the time from when the cooking device (100) starts operating until when it ends. The cooking device (100) may end the operation of the cooking device (100) based on the identification that the operating time has been reached. The cooking mode may refer to the cooking style of food. The cooking device (100) may determine the method of heat transfer or the method of controlling internal airflow according to the cooking mode. The cooking mode may include, for example, convection, bake, variable broil, steam bake, steam roast, convection vegetable, air fry, and air sous vide.

[0061] The cooking chamber (20) includes a space for accommodating food ingredients and may be a space for performing cooking operations on the food ingredients. The cooking chamber (20) may include a heating unit for generating heat.

[0062] According to one embodiment, the cooking device (100) can photograph food located in the cooking room (20) using a camera (130). The camera (130) may be positioned to photograph the interior of the cooking room (20). For example, the camera (130) may be positioned to photograph food ingredients from the front. Accordingly, the image obtained through the camera (130) may be a top-view image or an image taken at an angle close to a top-view image.

[0063] According to one embodiment, the electronic device (200) may be one of various types of electronic devices, such as a laptop, a smartphone, a tablet, or a cellular phone. These electronic devices (200) are merely examples, and the electronic device (200) may include, but is not limited to, a personal computer, a terminal, a portable telephone, a handheld device, a wearable device (e.g., a smart watch).

[0064] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0065] The user device (2) of FIG. 1 may correspond to the electronic device (200) of FIG. 3. The electronic device (200) of FIG. 3 may include the user device (2) of FIG. 1. Referring to FIG. 3, the electronic device (200) may include at least one processor (210) (hereinafter referred to as processor (110)), at least one memory (220) (hereinafter referred to as memory (220)), at least one communication circuit (230) (hereinafter referred to as communication circuit (230)), and at least one display (240) (hereinafter referred to as display (240)).

[0066] The components illustrated in FIG. 3 are merely exemplary. For example, the electronic device (200) may include other components (e.g., at least one image sensor, at least one sensor, a PMIC (power management integrated circuitry), an audio processing circuit, an antenna, a rechargeable battery, and / or an input / output interface). For example, some components may be omitted from the electronic device (200). For example, some components may be integrated into a single component.

[0067] The processor (210) can control the overall operation of the electronic device (200). Specifically, the processor (210) can control the overall operation of the electronic device (200) by being connected to each component of the electronic device (200). The processor (210) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. The processor (210) can control one or any combination of other components of the electronic device (200) and can perform operations or data processing related to communication. The processor (210) can execute one or more programs or instructions stored in the memory (220) of the electronic device (200). For example, the processor (210) can perform the method according to one embodiment of the present disclosure by executing one or more instructions stored in memory (220).

[0068] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0069] The processor (210) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When the processor (210) is implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. Additionally, each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0070] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0071] The memory (220) can store data necessary for various embodiments. Depending on the purpose of data storage, the memory (220) may be implemented in the form of a memory embedded in the electronic device (200) or in the form of a memory that can be attached to and detached from the electronic device (200). For example, data for operating the electronic device (200) may be stored in a memory embedded in the electronic device (200), and data for the expansion function of the electronic device (200) may be stored in a memory that can be attached to and detached from the electronic device (200). Meanwhile, the memory embedded in the electronic device (200) may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), etc.), non-volatile memory (e.g., OTPROM (one-time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash, etc.), hard drive, or solid state drive (SSD). Additionally, the memory that is detachable from the electronic device (200) may be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory connectable to a USB port (e.g., USB memory), etc. there is.

[0072] Instructions may be stored in the memory (220). The processor (210) may perform the operation of the electronic device (200) according to various embodiments of the present disclosure by executing the instructions stored in the memory (220) individually or collectively. Additionally, programs and data for operating the electronic device (200) may be stored in the memory (220). For example, the memory (220) may store one or more software applications, such as operating system (or system) software applications, firmware software applications, driver software applications, plugin (e.g., add-in, add-on, and / or applet) software applications, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (210). For example, memory (220) can store instructions that can be called by an API (application programming interface). For example, memory (120) can store instructions within a library.

[0073] In the above-described embodiment, various data is described as being stored in the external memory (220) of the processor (210), but at least some of the above-described data may be stored in the internal memory of the processor (210) according to at least one implementation example of the electronic device (200) or the processor (210).

[0074] In the embodiments of the present disclosure, a processor may mean a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto.

[0075] The electronic device (200) can communicate with an external device (e.g., a server device and / or an external device) through a communication circuit (230). The processor (210) can receive various data or information from an external device connected through the communication circuit (230) and can transmit various data or information to the external device.

[0076] The communication circuit (230) can communicate with an external device through a nearby access point (AP). The access point (AP) can connect the local area network (LAN) to which the electronic device (200) is connected to a wide area network (WAN) to which the external device is connected. The electronic device (200) can be connected to the external device through the network (WAN). Additionally, the communication circuit (230) can perform device-to-device (D2D) communication with the external device. For example, the communication circuit (230) can communicate with the external device over short distances without using an access point.

[0077] The communication circuit (230) can communicate with an external device using various types of communication methods. For example, the communication circuit (230) may include a LAN communication module such as an Ethernet module. The communication circuit (230) may include wireless communication modules such as Wi-Fi, Wi-Fi Direct, Bluetooth, BLE (Bluetooth Low Energy), Zigbee, NFC, Z-Wave, and infrared communication. The communication circuit (230) may include cellular communication modules such as 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), and 5G. The communication circuit (230) may include communication modules such as HDMI (High-Definition Multimedia Interface) and USB (Universal Serial Bus).

[0078] The display (240) may be implemented as a display including a self-emissive element or as a display including a non-emissive element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, or a QLED (Quantum dot light-emitting diodes). The display (240) may also include a driving circuit, a backlight unit, etc., which may be implemented in the form of an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. According to one example, the display (240) may be implemented as a flat display, a curved display, a folding or / and rolling flexible display, etc.

[0079] FIG. 4 is a block diagram illustrating the configuration of a cooking device according to one embodiment.

[0080] At least one of the electric range (13), electric oven (14), or microwave oven (19) of FIG. 1 may correspond to the cooking device (100) of FIG. 3. The cooking device (100) of FIG. 3 may include at least one of the electric range (13), electric oven (14), or microwave oven (19) of FIG. 1. Referring to FIG. 4, the cooking device (100) may include at least one processor (110) (hereinafter referred to as processor (110)), at least one memory (120) (hereinafter referred to as memory (120)), at least one camera (130) (hereinafter referred to as camera (130)), and at least one communication circuit (140) (hereinafter referred to as memory (140)).

[0081] The processor (110) can control the overall operation of the cooking device (100). Specifically, the processor (110) is connected to each component of the cooking device (100) to control the overall operation of the cooking device (100). The processor (110) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. The processor (110) can control one or any combination of other components of the cooking device (100) and can perform operations or data processing related to communication. The processor (110) can execute one or more programs or instructions stored in the memory (120) of the cooking device (100). For example, the processor (110) can perform the method according to one embodiment of the present disclosure by executing one or more instructions stored in memory (120).

[0082] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0083] The processor (110) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When the processor (110) is implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. Additionally, each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0084] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0085] The memory (120) can store data necessary for various embodiments. Depending on the purpose of data storage, the memory (120) may be implemented in the form of a memory embedded in the cooking device (100) or in the form of a memory that can be attached to and detached from the cooking device (100). For example, data for operating the cooking device (100) may be stored in a memory embedded in the cooking device (100), and data for the expansion function of the cooking device (100) may be stored in a memory that can be attached to and detached from the cooking device (100). Meanwhile, the memory embedded in the cooking device (100) may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), etc.), non-volatile memory (e.g., OTPROM (one-time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash, etc.), hard drive, or solid state drive (SSD). Additionally, the memory that is detachable from the cooking device (100) may be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory connectable to a USB port (e.g., USB memory), etc. there is.

[0086] Instructions may be stored in the memory (120). The processor (110) may perform the operation of the cooking device (100) according to various embodiments of the present disclosure by executing the instructions stored in the memory (120) individually or collectively. Additionally, programs and data for operating the cooking device (100) may be stored in the memory (120). For example, the memory (120) may store one or more software applications, such as operating system (or system) software applications, firmware software applications, driver software applications, plugin (e.g., add-in, add-on, and / or applet) software applications, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (110). For example, memory (120) can store instructions that can be called by an API (application programming interface). For example, memory (120) can store instructions within a library.

[0087] In the above-described embodiment, various data is described as being stored in the external memory (120) of the processor (110), but at least some of the above-described data may be stored in the internal memory of the processor (110) according to at least one implementation example of the cooking device (100) or the processor (110).

[0088] In the embodiments of the present disclosure, a processor may mean a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto.

[0089] A camera (130) is provided inside a cooking room (20) and can photograph the inside of the cooking room (20). The camera (130) can photograph food contained inside the cooking room (20). The camera (130) can acquire an image corresponding to the subject by converting light emitted or reflected and transmitted from the subject into an electrical signal using an image sensor. According to one embodiment, the image sensor may include, for example, one image sensor selected from image sensors with different attributes such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes.

[0090] The communication circuit (140) can communicate with an external device through a nearby access point (AP). The access point (AP) can connect the local area network (LAN) to which the electronic device (200) is connected to a wide area network (WAN) to which the external device is connected. The electronic device (200) can be connected to the external device through the network (WAN). Additionally, the communication circuit (140) can perform device-to-device (D2D) communication with the external device. For example, the communication circuit (140) can communicate with the external device over short distances without using an access point.

[0091] The communication circuit (140) can communicate with an external device using various types of communication methods. For example, the communication circuit (140) may include a LAN communication module such as an Ethernet module. The communication circuit (140) may include wireless communication modules such as Wi-Fi, Wi-Fi Direct, Bluetooth, BLE (Bluetooth Low Energy), Zigbee, NFC, Z-Wave, and infrared communication. The communication circuit (140) may include cellular communication modules such as 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), and 5G. The communication circuit (140) may include communication modules such as HDMI (High-Definition Multimedia Interface) and USB (Universal Serial Bus).

[0092] FIG. 5 is a block diagram illustrating the detailed configuration of a cooking device according to one embodiment.

[0093] Referring to FIG. 5, the cooking device (100) may include a processor (110), a memory (120), a camera (130), a communication circuit (140), a display (150), a heating unit (160), and an interface (170). However, such a configuration is exemplary, and it is understood that in carrying out the present disclosure, new configurations may be added or some configurations may be omitted in addition to such configurations. Meanwhile, detailed descriptions of configurations shown in FIG. 5 that overlap with configurations shown in FIG. 4 will be omitted.

[0094] The display (150) may be implemented as a display including a self-emissive element or as a display including a non-emissive element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, or a QLED (Quantum dot light-emitting diodes). The display (150) may also include a driving circuit, a backlight unit, etc., which may be implemented in the form of an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. According to one example, the display (150) may be implemented as a flat display, a curved display, a folding or / and rolling flexible display, etc.

[0095] The heating unit (160) may be configured to heat the temperature inside the cooking chamber (20) to a cooking temperature set by the user. For example, a processor (110) may be electrically connected to the heating unit (160) and control the operation of the heating unit (160) based on operation information. The heating unit (160) may include an upper heating element, a lower heating element, a convection heating element, a grill heating element, etc.

[0096] An interface (170) is a configuration created to interact between two or more systems, devices, programs, or users. The interface (170) may include at least one of an input / output interface (160, 170).

[0097] The input interface (160) may include a circuit. The input interface (160) may receive user input for controlling the operation of the cooking device (100) and provide the user input to the processor (110). The input interface may include, for example, a tact switch, a push switch, a slide switch, a toggle switch, a micro switch, a touch switch, a touch pad, a touch screen, a jog dial, and / or a microphone.

[0098] The input interface (171) may be provided in an area of ​​the outer wall surface of the cooking device (100). For example, the input interface (171) may be provided in an area of ​​the front portion of the housing. However, the present disclosure is not limited thereto, and the input interface (171) may be provided in various locations, such as the front and / or side of the housing.

[0099] The output interface (172) includes a circuit, and the processor (110) can visually or audibly convey various information related to the cooking device (100) to the user through the output interface (172). The output interface (172) may include a Liquid Crystal Display (LCD) panel, a Light Emitting Diode (LED) panel, a speaker, etc.

[0100] FIG. 6 is a flowchart illustrating an example of a method in which a cooking device provides a notification about food ingredients inside a cooking chamber according to one embodiment.

[0101] The processor (110) of the cooking device (100) can perform at least one of the operations of FIG. 6. When the instructions stored in the memory (120) of the cooking device (100) are executed by the processor (110) of the cooking device (100), the cooking device (100) can perform the operations of FIG. 6.

[0102] In operation 610 of FIG. 6, according to one embodiment, the cooking device (100) can identify at least one food ingredient in the cooking chamber (20) based on an image (hereinafter referred to as the first image) obtained through a camera (130). In the present disclosure, the food ingredient may refer to the ingredients of a dish located in the cooking chamber (20). For example, let us assume that the dish is a steak. The food ingredient may include raw meat (e.g., chicken, beef, pork) and vegetables (e.g., potatoes, onions, asparagus). The operation of identifying at least one food ingredient based on the first image is described in detail in FIG. 7.

[0103] FIG. 7 is a drawing illustrating an example of an operation in which a cooking device (100) identifies food ingredients based on a first image according to one embodiment.

[0104] According to one embodiment, a cooking device (100) can identify at least one food ingredient by inputting a first image (700) into an artificial intelligence model (hereinafter referred to as the first artificial intelligence model). The first artificial intelligence model may include a deep learning model trained to identify food ingredients contained in the first image (700). Identifying food ingredients by the cooking device (100) may include identifying the type of food ingredient and the area (710-1 to 710-4) where the food ingredient is located within the first image (700).

[0105] According to one embodiment, the first artificial intelligence model may be a model trained using an image containing food ingredients, information about the region of the food ingredients within the image, and the type of food ingredients as input data, and information about the region of the food ingredients within the image and the type of food ingredients as output data. The fact that an artificial intelligence model is trained may mean that a basic artificial intelligence model (e.g., an artificial intelligence model including arbitrary random parameters) is trained using a plurality of training data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). Such training may be performed through a separate server and / or system, but is not limited thereto, and may also be performed in a cooking device (100). Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0106] According to one embodiment, the first artificial intelligence model may be implemented in the form of an on-device stored in memory (120), but is not necessarily limited thereto. For example, the first artificial intelligence model may be stored on an external server, and the processor (110) may receive information regarding the type of expected food ingredient and the area of ​​the food ingredient after sending a prompt to the external server. For example, the prompt may include a first image.

[0107] Referring to FIG. 7, the cooking device (100) can input a first image (700) into a first artificial intelligence model to obtain the type of food ingredient and at least one area (710-1 to 710-4) corresponding to the food ingredient. For example, the cooking device (100) can use the first artificial intelligence model to identify beef located in the first area (710-1) (hereinafter referred to as the first beef), beef located in the second area (710-2) (hereinafter referred to as the second beef), potatoes located in the third area (710-3), and asparagus located in the fourth area (710-4).

[0108] In operation 620 of FIG. 6, according to one embodiment, when a food ingredient (hereinafter referred to as the first food ingredient) is selected from at least one food ingredient based on user input, the cooking device (100) may acquire a plurality of predicted images. According to one embodiment, the plurality of predicted images may include a plurality of predicted images of the cooking state of the first food ingredient expected at a plurality of points within the cooking time as the cooking device (100) performs a cooking operation according to operation information (e.g., cooking temperature, cooking mode, cooking time). The operation of the cooking device (100) identifying the first food ingredient based on user input is described in detail below in FIG. 8.

[0109] According to one embodiment, a cooking device (100) can obtain a plurality of predicted images of a first food ingredient by inputting a prompt into an artificial intelligence model (hereinafter referred to as a second artificial intelligence model). The prompt may include a prompt for requesting predicted images of the first food ingredient corresponding to a plurality of time points.

[0110] According to one embodiment, the cooking device (100) can generate a prompt in a rule-based manner. The cooking device (100) can generate a prompt requesting the creation of an image related to a notification by inserting keywords (e.g., cooking time, cooking temperature, cooking mode, and type of food ingredients, multiple points in time) into placeholders of a prompt template. For example, a placeholder may be a location in the prompt template where keywords are inserted. Placeholders to which keywords are mapped may be designated according to the attributes of the keywords. The cooking device (100) can generate a prompt by inserting keywords into placeholders of a prompt template. Rules for the cooking device (100) to generate a prompt based on a first food ingredient image, cooking time, cooking temperature, cooking mode, type of first food ingredients, and multiple points in time may be stored in memory (120).

[0111] For example, the cooking device (100) can generate a prompt by inserting an image of the first food ingredient, the type of the first food ingredient, operation information, and multiple times into a prompt template such as “a photo of Delicious-looking {type of the first food ingredient and image of the first food ingredient} cooked in {cooking mode} mode set at {multiple times} minutes and {cooking temperature} degrees”.

[0112] According to one embodiment, the cooking device (100) can input a generated prompt into a second artificial intelligence model to obtain a plurality of predicted images of a first food ingredient. The second artificial intelligence model may be a model trained using a plurality of images of the food ingredient before cooking, images of the food ingredient at multiple points in time, and information on the type and operation of the food ingredient as input data, and a predicted image of the food ingredient at a specific point in time as output data. For example, the first artificial intelligence model may be a generative adversarial network (GAN) or a Diffusion-based generative model using a Transformer structure.

[0113] According to one embodiment, the second artificial intelligence model may be stored in memory (120), but is not necessarily limited thereto. For example, the second artificial intelligence model may be stored on an external server, and the cooking device (100) may receive information about a plurality of expected images after sending a prompt to the external server.

[0114] According to one embodiment, the operation of the cooking device (100) acquiring a plurality of predicted images may be performed frequently or periodically. For example, the cooking device (100) may receive user input to select the first food ingredient among at least one food ingredient in the cooking chamber after the cooking operation is stopped based on user input. When the cooking device (100) performs the cooking operation again, the cooking device (100) may acquire a plurality of predicted images of the first food ingredient based on the cooking status of the first food ingredient and the remaining cooking time.

[0115] FIG. 8 is a diagram illustrating an example of an operation in which a cooking device (100) identifies a first food ingredient among at least one food ingredient based on user input, according to one embodiment.

[0116] According to one embodiment, the cooking device (100) can identify a first food ingredient among at least one food ingredient based on user input received from an electronic device (200) through a communication circuit (140).

[0117] According to one embodiment, a cooking device (100) can transmit information regarding a first image, a type of food ingredient (e.g., first beef, second beef, potato, asparagus), and areas corresponding to the food ingredient (810-1 to 810-4) to an electronic device (200). However, it is not limited thereto, and the cooking device (100) can transmit information regarding the first image, the type of the first food ingredient, and the areas corresponding to the first food ingredient to the electronic device (200) through a server (e.g., the server (3) of FIG. 1). The electronic device (200) can be connected to the server via a network.

[0118] According to one embodiment of 801 in FIG. 8, the electronic device (200) may receive user input for selecting a first food ingredient among at least one food ingredient. The user input may include, for example, a touch input by the user for an area (810-1) corresponding to the first food ingredient.

[0119] According to one embodiment of 802 in FIG. 8, when the electronic device (200) receives user input for a first food ingredient, it may create a border of an area (820) corresponding to the first food ingredient or insert a shade into an area (820) corresponding to the first food ingredient. However, it is not limited thereto, and the electronic device (200) may display an effect on the display (240) indicating that the user has selected an area (820) corresponding to the first food ingredient.

[0120] According to one embodiment, the electronic device (200) can transmit user input regarding the first food ingredient to the cooking device (100). For example, the user input regarding the first food ingredient may include information for identifying an area (820) corresponding to the first food ingredient (e.g., first beef) or coordinate information where the user's touch input is detected. The cooking device (100) can identify the first food ingredient based on the user input received from the electronic device (200).

[0121] Meanwhile, FIG. 8 illustrates a case where the cooking device (100) identifies a first food ingredient based on user input received from the electronic device (200), but is not limited thereto. For example, the cooking device (100) may display a first image and areas (810-1 to 810-4) corresponding to the food ingredient on a display (150) mounted on the cooking device (100). The cooking device (100) may detect a user touch input for the area (810-1) corresponding to the first food ingredient among the areas (810-1 to 810-4) corresponding to the food ingredients. When user input for the area (810-1) corresponding to the first food ingredient is detected, the cooking device (100) may select the first food ingredient.

[0122] According to one embodiment of operation 630 of FIG. 6, the cooking device (100) may receive user input for selecting a predicted image among a plurality of predicted images. The operation of the cooking device (100) selecting a predicted image based on the user input is described in detail in FIG. 9.

[0123] FIG. 9 is a diagram illustrating an example of an operation in which a cooking device (100) selects a predicted image from a plurality of predicted images based on user input, according to one embodiment.

[0124] In 901 of FIG. 9, according to one embodiment, an electronic device (200) may receive user input for selecting an expected image (940) from a plurality of expected images (920-1 to 920-5). According to one embodiment, a cooking device (100) may transmit a plurality of expected images (920-1 to 920-5) to the electronic device (200) through a communication circuit (140). The electronic device (200) may display the plurality of expected images (920-1 to 920-5) received from the cooking device (100) on a display (240). The electronic device (200) may display a progress bar (930) at the bottom of the plurality of expected images (920-1 to 920-5). The progress bar (930) may be a UI element corresponding to the cooking time of the cooking device (100). For example, the left end of the progress bar (930) may correspond to the time when the cooking device (100) starts operating, and the right end of the progress bar (930) may correspond to the time when the cooking device (100) ends operating according to the cooking time. The electronic device (200) may display a plurality of predicted images (920-1 to 920-5) at points corresponding to each predicted image on the progress bar (930).

[0125] In 902 of FIG. 9, according to one embodiment, an electronic device (200) can detect a user's touch input on a progress bar (950). The electronic device (200) can identify an expected image (940) corresponding to the point where the user's touch input was detected. However, it is not limited thereto, and when the electronic device (200) detects a user's touch input on the progress bar (950), it can acquire an expected image for the point where the touch input was received. The electronic device (200) can display the acquired expected image.

[0126] According to one embodiment, the cooking device (100) may receive user input for selecting at least one of a plurality of predicted images (920-1 to 920-5). For example, the cooking device (100) may receive user input for selecting a first predicted image (e.g., 920-2) and a second predicted image (e.g., 920-4) among a plurality of predicted images (920-1 to 920-5).

[0127] Meanwhile, the UI for selecting an expected image (940) from a plurality of expected images (920-1 to 920-5) is not limited to a progress bar. For example, the UI for selecting an expected image (940) from a plurality of expected images (920-1 to 920-5) may include a UI for guiding the selection of at least one of a plurality of options, such as a list box, a button group, or a radio button.

[0128] FIG. 9 illustrates a case where the cooking device (100) identifies a predicted image based on user input received from the electronic device (200), but is not limited thereto. For example, the cooking device (100) may display a plurality of predicted images (920-1 to 920-5) on a display (150) mounted on the cooking device (100). The cooking device (100) may detect a user touch input for a predicted image (940) among the plurality of predicted images (920-1 to 920-5). When user input for a predicted image (940) is detected, the cooking device (100) may select the predicted image (940).

[0129] In operation 640 of FIG. 6, according to one embodiment, the cooking device (100) can obtain an image (hereinafter referred to as the second image) by using a camera (130) to photograph the cooking chamber (20) while the cooking device (100) performs a cooking operation.

[0130] According to one embodiment, the cooking device (100) can acquire a second image using a camera (130) at preset time intervals from the time the cooking device (100) starts operating. For example, if the cooking time of the cooking device (100) is 40 minutes, the cooking device (100) can acquire a plurality of second images by photographing the cooking room (20) at 1-minute intervals.

[0131] In operation 650 of FIG. 6, according to one embodiment, if the cooking device (100) identifies that the cooking state of the first food ingredient in the second image corresponds to at least one expected image, it can transmit a notification about the first food ingredient to an external electronic device through a communication circuit (140).

[0132] According to one embodiment, the cooking device (100) can identify that the cooking state of the first food ingredient in the second image corresponds to the expected image when the similarity between the food ingredient image (hereinafter referred to as the first food ingredient image) including the first food ingredient and the expected image is greater than or equal to a preset value.

[0133] According to one embodiment, the cooking device (100) can obtain a first food ingredient image from a second image. For example, the cooking device (100) can obtain a first food ingredient image by inputting the second image into a first artificial intelligence model.

[0134] According to one embodiment, the cooking device (100) can identify the similarity between a first food ingredient image and an expected image. For example, the cooking device (100) can calculate the similarity based on a color vector for the first food ingredient image and a color vector for the expected image. A color vector may refer to a vector in which the average value of RGB values ​​for a plurality of pixels constituting an image is a component. For example, the cooking device (100) can obtain an average value for each color by summing the values ​​for red (R), green (G), and blue (B) of a plurality of pixels included in the image and then dividing by the number of pixels included in the image. The cooking device (100) can obtain a color vector in which the average value for each color is a component of the vector.

[0135] According to one embodiment, the cooking device (100) can obtain a first color vector corresponding to a first food ingredient image. For example, the cooking device (100) can calculate the average of color vectors for a plurality of pixels constituting the first food ingredient image. The cooking device (100) can identify the average of the plurality of color vectors as the first color vector corresponding to the first food ingredient image.

[0136] According to one embodiment, the cooking device (100) can obtain a second color vector corresponding to an expected image. For example, the cooking device (100) can calculate the average of color vectors for a plurality of pixels constituting the expected image. The cooking device (100) can identify the average of the plurality of color vectors as the second color vector corresponding to the expected image.

[0137] According to one embodiment, the cooking device (100) can identify the similarity between a first color vector and a second color vector. For example, the cooking device (100) can obtain the similarity between the first color vector and the second color vector based on the cosine similarity between the first color vector and the second color vector. In this case, the closer the cosine similarity between the first color vector and the second color vector is to 1, the more similar the two images are.

[0138] However, the method for determining the similarity between the first color vector and the second color vector is not limited to the examples described above. For instance, the method for determining the similarity between the first color vector and the second color vector may include, but is not limited to, pixel-based methods (e.g., MSE (mean squared error), PSNR (peak signal-to-noise ratio) or structural similarity determination methods (e.g., SSIM (structural similarity index)).

[0139] According to one embodiment, when the cooking device (100) identifies that the cooking state of the first food ingredient in the second image corresponds to the expected image, it can transmit a notification about the first food ingredient to an external electronic device through the communication circuit (140).

[0140] According to one embodiment, the external electronic device may include a home appliance or a user terminal device placed (or located) within an environment such as a house. The home appliance may be at least one of various types of home appliances. For example, the home appliance may include at least one of an oven, a smart television, a speaker, an electric range, and an electric oven, but is not limited thereto. The user terminal device may be at least one of various types of terminal devices. For example, the user terminal device may include at least one of a smartphone, a tablet, a smart watch, a smart ring, and a wireless earphone, but is not limited thereto.

[0141] According to one embodiment, the cooking device (100) may determine an external electronic device to transmit a notification regarding a first food ingredient based on user input. The user input may include, for example, at least one of information about the external electronic device received from the electronic device (200) or a user's touch input for selecting the external electronic device from the cooking device (100). The cooking device (100) may transmit a notification regarding the food ingredient to the determined external electronic device.

[0142] According to one embodiment, the cooking device (100) may provide a notification regarding the ingredients when a time set by the user arrives. The time set by the user may refer to a time when the state of the first ingredients in the second image obtained using the camera (130) corresponds to the expected image selected by the user.

[0143] FIG. 10 is a drawing illustrating an example in which a cooking device, according to one embodiment, provides a notification of a first food ingredient to an external electronic device.

[0144] Referring to FIG. 10, when the cooking device (100) identifies that the cooking state of the first food ingredient in the second image corresponds to the expected image, it may transmit a notification regarding the first food ingredient to an external electronic device (e.g., a user terminal device). The notification regarding the first food ingredient may include at least one of a notification via UI (1010) or a notification via sound (1020).

[0145] For example, the cooking device (100) can transmit information to an external electronic device to notify that a preset event (e.g., the arrival of a time selected by the user) has occurred. For example, the external electronic device can display a UI (1010) containing information to notify that a preset event has occurred on a display. For example, the external electronic device can output a sound (e.g., “Ding,” “Please flip the ingredients,” etc.) through a speaker.

[0146] FIG. 11 is a drawing illustrating an example of a method in which a cooking device provides notifications about a plurality of food ingredients inside a cooking chamber, according to one embodiment.

[0147] Referring to 1101 in FIG. 11, according to one embodiment, a cooking device (100) can identify a first food ingredient (1110) and a second food ingredient (1120) based on user input among a plurality of food ingredients identified in a first image. Since the operation of identifying food ingredients based on user input has been described above, redundant details are omitted.

[0148] According to one embodiment, the cooking device (100) may display the number of notifications set on the first food ingredient (1110) in an area including the first food ingredient. The number of notifications set on the first food ingredient (1110) may be the number of predicted images selected based on user input. For example, if the user selects one of the multiple predicted images of the first food ingredient (1110), the number of notifications set on the first food ingredient (1110) may be one.

[0149] Referring to 1102 in FIG. 11, according to one embodiment, a cooking device (100) can identify whether the cooking state of the first food ingredient (1110) and the second food ingredient (1120) in the second image corresponds to the expected image for the first food ingredient and the expected image for the second food ingredient, respectively.

[0150] According to one embodiment, when the cooking device (100) identifies a second food ingredient (1120) among a plurality of food ingredients based on user input, it may display (1130) a plurality of predicted images and a progress bar corresponding to the cooking state of the second food ingredient on a display (150). The cooking device (100) may select a predicted image among a plurality of predicted images of the second food ingredient (1120) based on user input.

[0151] According to one embodiment, when the cooking device (100) identifies that the cooking state of the first food ingredient (1110) in the second image corresponds to an expected image of the first food ingredient, it can transmit a notification regarding the first food ingredient (1110) to the external electronic device through the communication circuit (140).

[0152] According to one embodiment, if the cooking state of the second food ingredient (1120) in the second image is identified as corresponding to the expected image of the second food ingredient, a notification regarding the second food ingredient (1120) can be transmitted to an external electronic device through the communication circuit (140).

[0153] FIG. 12 is a diagram illustrating an example of a method in which a cooking device provides a notification about all food ingredients inside a cooking chamber according to one embodiment.

[0154] Referring to 1201 in FIG. 12, the electronic device (200) can display a first image, a type of food ingredient, and areas (1210-1 to 1210-n) corresponding to the food ingredient on a display (240). The electronic device (200) can receive information regarding the first image, the type of food ingredient, and areas (1210-1 to 1210-n) corresponding to the food ingredient from a cooking device (100).

[0155] Referring to 1202 in FIG. 12, the electronic device (200) can receive user input for selecting all ingredients. User input for selecting all ingredients may include, but is not limited to, input of touching the display (240) multiple times in succession.

[0156] Referring to 1203 in FIG. 12, when the electronic device (200) receives user input for selecting all ingredients, it may insert shading in the areas (1210-1 to 1210-n) corresponding to all ingredients. Referring to 1204 in FIG. 12, the electronic device (200) may receive user input for selecting any one ingredient among all ingredients (e.g., a first ingredient). User input for selecting the first ingredient among all ingredients may include, for example, a user input of long-pressing the area (1230-3) corresponding to the first ingredient among the areas (1230-1 to 1230-n) corresponding to all ingredients for a preset time or longer. According to one embodiment, the electronic device (200) may transmit user input regarding the first ingredient to the cooking device (100).

[0157] Referring to 1205 in FIG. 12, according to one embodiment, an electronic device (200) may display a UI (1240) including a plurality of predicted images of a first food ingredient and a progress bar on a display (240). According to one embodiment, when a cooking device (100) receives user input regarding the first food ingredient, it may acquire a plurality of predicted images of the first food ingredient. The cooking device (100) may transmit the plurality of predicted images of the first food ingredient to the electronic device (200).

[0158] Referring to 1206 in FIG. 12, according to one embodiment, an electronic device (200) may receive user input for selecting a predicted image among a plurality of predicted images for a first food ingredient. The user input for selecting a predicted image among a plurality of predicted images may include user input for a UI (1250) including a progress bar. The electronic device (200) may transmit the user input to a cooking device.

[0159] According to one embodiment, if the cooking device (100) identifies that the cooking state of all ingredients in the second image corresponds to the expected image, it can transmit a notification about all ingredients to an external electronic device through a communication circuit (140).

[0160] According to one embodiment, the cooking device (100) can identify that the cooking state of all ingredients in the second image corresponds to the expected image when the similarity between a plurality of ingredient images including all ingredients and an expected image is greater than or equal to a preset value. Since the method for determining the similarity between images has been described in detail in FIG. 6, redundant content is omitted.

[0161] FIG. 13 is a diagram illustrating an example of a method in which a cooking device provides a notification regarding a food ingredient when the food ingredient is not recognized, according to one embodiment.

[0162] A case where the food ingredient (1310) is not recognized may mean that the cooking device (100) inputs the first image into the first artificial intelligence model and the output information (e.g., type of food ingredient, area corresponding to the food ingredient) does not include information about the food ingredient (1310).

[0163] Referring to 1301 in FIG. 13, according to one embodiment, an electronic device (200) may receive user input for obtaining a portion (1320) of a food ingredient. User input for obtaining a portion (1320) of a food ingredient may include user input for dragging a portion of a display (240). The electronic device (200) may receive voice input regarding the type of food ingredient from a user. For example, the electronic device (200) may receive input regarding the type of food ingredient (e.g., “bread”) through a microphone sensor. The electronic device (200) may display a UI including the type of food ingredient in a portion (1330). However, it is not limited thereto, and the electronic device (200) may, of course, receive text input regarding the type of food ingredient from a user. The electronic device (200) may transmit the portion (1320) of the food ingredient and the type of food ingredient received from the user to a cooking device (100).

[0164] Referring to 1302 in FIG. 13, according to one embodiment, a cooking device (100) can obtain a plurality of predicted images of a food ingredient (1310) by inputting a region (1320) of the food ingredient and the type of the food ingredient received from an electronic device (200) into a second artificial intelligence model. The electronic device (200) can display a UI (1340) including a plurality of predicted images (1310) of the food ingredient and a progress bar on a display (240). The electronic device (200) can receive user input to obtain a predicted image among the plurality of predicted images. The electronic device (200) can transmit the received user input to the cooking device (100). According to one embodiment, if the cooking device (100) identifies that the cooking state of a region (1330) within a second image corresponds to a predicted image, it can transmit a notification regarding the entire food ingredient to an external electronic device through a communication circuit (140).

[0165] According to one embodiment, the cooking device (100) can identify that the cooking state of a food ingredient (e.g., bread) corresponds to the expected image when the similarity between the color vector of a region (1330) and the color vector of the expected image is greater than or equal to a preset value. Since the method for determining similarity between images is described in detail in FIG. 6, redundant content is omitted.

[0166] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs.

[0167] The cooking device (100) can provide notifications through the display of the cooking device (100) or a terminal connected to the cooking device (100). The cooking device (100) allows the user to select ingredients that require separate notifications or control among the ingredients loaded into the cooking device. The user can receive a notification if the selected ingredients require intervention in the middle (e.g., 'flipping') or stop operation (e.g., 'Burn Notification'). Through this, the user does not need to continuously monitor the cooking device (100) or the terminal connected to the cooking device (100) for a long time, and can freely control the selection of ingredients and notification settings through the display, thereby increasing cooking efficiency.

[0168] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0169] As described above, according to one embodiment of the present disclosure, a cooking device including a cooking chamber may include at least one processor including a memory for storing camera communication circuit instructions for photographing the cooking chamber and a processing circuitry.

[0170] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device may identify at least one food ingredient in the cooking chamber based on a first image acquired through the camera, and when the first food ingredient is selected among the at least one food ingredient based on user input, acquire a plurality of predicted images corresponding to the cooking state of the first food ingredient expected according to the cooking operation of the cooking device, receive user input for selecting a predicted image among the plurality of predicted images, and while the cooking device performs the cooking operation, capture the cooking chamber using the camera to acquire a second image, and when the cooking state of the first food ingredient in the second image is identified as corresponding to the predicted image, transmit a notification regarding the first food ingredient to an external electronic device through the communication circuit.

[0171] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device may acquire an image of a food ingredient including the first food ingredient from the second image, identify a similarity between a first color vector corresponding to the image of the food ingredient and a second color vector corresponding to the predicted image, and if the similarity is identified as being greater than or equal to a threshold value, identify the cooking state of the first food ingredient within the second image as corresponding to the predicted image.

[0172] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device may calculate the average of color vectors for a plurality of pixels of the food ingredient image to obtain a first color vector corresponding to the food ingredient image, and calculate the average of color vectors for a plurality of pixels of the predicted image to obtain a second color vector corresponding to the predicted image.

[0173] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device acquires an image of a food ingredient including the first food ingredient in the first image, inputs the image of the food ingredient, the type of the first food ingredient, and operation information set in the cooking device into an artificial intelligence model to acquire the plurality of predicted images from the artificial intelligence model, and the operation information may include at least one of a cooking time, a cooking temperature, and a cooking mode.

[0174] For example, the plurality of predicted images may include a plurality of predicted images corresponding to the cooking state of the first food ingredient expected at a plurality of points within the cooking time as the cooking device performs the cooking operation based on the operation information.

[0175] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device receives user input for selecting the first food ingredient among at least one food ingredient in the cooking chamber after the cooking operation is stopped, and when the cooking device performs the cooking operation again, it may acquire a plurality of expected images of the first food ingredient based on the cooking status of the first food ingredient and the remaining cooking time.

[0176] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device may identify whether the cooking state of the first food ingredient and the second food ingredient within the second image corresponds to the expected image for the first food ingredient and the expected image for the second food ingredient, respectively, when the first food ingredient and the second food ingredient are selected based on user input among a plurality of food ingredients identified in the first image, and when the cooking state of the first food ingredient within the second image is identified as corresponding to the expected image for the first food ingredient, transmit a notification regarding the first food ingredient to the external electronic device through the communication circuit, and when the cooking state of the second food ingredient within the second image is identified as corresponding to the expected image for the second food ingredient, transmit a notification regarding the second food ingredient to the external electronic device through the communication circuit.

[0177] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device may receive user input for selecting a first predicted image and a second predicted image among a plurality of predicted images for the first food ingredient, and if the cooking state of the first food ingredient in the second image is identified as corresponding to the first predicted image for the first food ingredient, transmit a first notification for the first food ingredient to the external electronic device through the communication circuit, and if the cooking state of the first food ingredient in the second image is identified as corresponding to the second predicted image for the first food ingredient, transmit a second notification for the first food ingredient to the external electronic device through the communication circuit.

[0178] For example, when the above instructions are executed individually or collectively by the at least one processor, the cooking device may acquire a plurality of predicted images corresponding to the entire food ingredient expected according to the cooking operation of the cooking device based on receiving user input for selecting the entire food ingredient in the cooking chamber, receive user input for selecting a predicted image among the plurality of predicted images, and if the cooking state of the entire food ingredient in the second image is identified as corresponding to the predicted image, transmit a notification regarding the first food ingredient to an external electronic device through the communication circuit.

[0179] According to one embodiment of the present disclosure, a method for controlling a cooking device including a cooking chamber comprises the step of identifying at least one food ingredient in the cooking chamber based on a first image obtained through a camera included in the cooking device;

[0180] For example, when a first food ingredient is selected based on user input among the at least one food ingredient, the method may include the steps of: acquiring a plurality of predicted images corresponding to the cooking state of the first food ingredient expected according to the cooking operation of the cooking device; receiving user input for selecting a predicted image among the plurality of predicted images; acquiring a second image by photographing the cooking room using the camera while the cooking device performs the cooking operation; and, when the cooking state of the first food ingredient in the second image is identified as corresponding to the predicted image, transmitting a notification regarding the first food ingredient to an external electronic device.

[0181] For example, the step of transmitting a notification regarding the first food ingredient to an external electronic device may include the step of obtaining a food ingredient image containing the first food ingredient from the second image, the step of identifying a similarity between a first color vector corresponding to the food ingredient image and a second color vector corresponding to the predicted image, and, if the similarity is identified to be greater than or equal to a threshold value, the step of identifying that the cooking state of the first food ingredient within the second image corresponds to the predicted image.

[0182] For example, the step of transmitting a notification regarding the first food ingredient to an external electronic device may include the step of calculating the average of color vectors for a plurality of pixels of the food ingredient image to obtain a first color vector corresponding to the food ingredient image, and the step of calculating the average of color vectors for a plurality of pixels of the predicted image to obtain a second color vector corresponding to the predicted image.

[0183] For example, the step of acquiring the plurality of predicted images may include the step of acquiring an image of a food ingredient including the first food ingredient in the first image, and the step of acquiring the plurality of predicted images from the artificial intelligence model by inputting the image of the food ingredient, the type of the first food ingredient, and operation information set in the cooking device into the artificial intelligence model.

[0184] For example, the above operation information may include at least one of cooking time, cooking temperature, and cooking mode.

[0185] For example, the plurality of predicted images may include a plurality of predicted images corresponding to the cooking state of the first food ingredient expected at a plurality of points within the cooking time as the cooking device performs the cooking operation based on the operation information.

[0186] For example, the step of acquiring the plurality of predicted images may include receiving user input to select the first food ingredient among at least one food ingredient in the cooking chamber after the cooking operation is stopped, and when the cooking device performs the cooking operation again, acquiring the plurality of predicted images for the first food ingredient based on the cooking status of the first food ingredient and the remaining cooking time.

[0187] For example, the control method may further include the steps of: identifying whether the cooking state of the first food ingredient and the second food ingredient within the second image corresponds to the expected image of the first food ingredient and the expected image of the second food ingredient, respectively, when the first food ingredient and the second food ingredient are selected based on user input among a plurality of food ingredients identified in the first image; transmitting a notification regarding the first food ingredient to the external electronic device through the communication circuit when it is identified that the cooking state of the first food ingredient within the second image corresponds to the expected image of the first food ingredient; and transmitting a notification regarding the second food ingredient to the external electronic device through the communication circuit when it is identified that the cooking state of the second food ingredient within the second image corresponds to the expected image of the second food ingredient.

[0188] For example, the control method may further include the steps of: receiving user input for selecting a first predicted image and a second predicted image among a plurality of predicted images for the first food ingredient; transmitting a first notification for the first food ingredient to the external electronic device through the communication circuit when the cooking state of the first food ingredient within the second image is identified as corresponding to the first predicted image for the first food ingredient; and transmitting a second notification for the first food ingredient to the external electronic device through the communication circuit when the cooking state of the first food ingredient within the second image is identified as corresponding to the second predicted image for the first food ingredient.

[0189] For example, the control method may further include the steps of: acquiring a plurality of predicted images corresponding to the entire food ingredient expected according to the cooking operation of the cooking device based on receiving user input for selecting the entire food ingredient in the cooking chamber; receiving user input for selecting a predicted image among the plurality of predicted images; and, if the cooking state of the entire food ingredient within the second image is identified as corresponding to the predicted image, transmitting a notification regarding the first food ingredient to an external electronic device through the communication circuit.

[0190] According to one embodiment of the present disclosure, a non-transient computer-readable recording medium stores one or more instructions executed by a processor of a cooking device to perform an operation, wherein the operation may include: identifying at least one food ingredient in the cooking chamber based on a first image acquired through a camera included in the cooking device; when the first food ingredient is selected among the at least one food ingredient based on user input, acquiring a plurality of predicted images corresponding to the cooking state of the first food ingredient expected according to the cooking operation of the cooking device; receiving user input for selecting a predicted image among the plurality of predicted images; capturing the cooking chamber using the camera while the cooking device performs a cooking operation to acquire a second image; and when the cooking state of the first food ingredient in the second image is identified as corresponding to the predicted image, transmitting a notification regarding the first food ingredient to an external electronic device.

[0191] Although various embodiments have been described above, each embodiment is not necessarily implemented individually, and may be combined with at least one other embodiment, either wholly or partially, to be implemented together in a single product.

[0192] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include other data of modulated data signals, such as computer-readable instructions, data structures, or program modules.

[0193] Additionally, computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage media' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, 'non-transitory storage media' may include a buffer in which data is stored temporarily.

[0194] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0195] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0196] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.

Claims

1. In a cooking device including a cooking chamber, A camera for photographing the above-mentioned kitchen; Communication circuit; Memory for storing instructions; and at least one processor including processing circuitry; and When the above instructions are executed individually or collectively by the at least one processor, the cooking device, Identifying at least one food ingredient in the cooking room based on a first image acquired through the camera, When a first food ingredient is selected based on user input among the above at least one food ingredient, a plurality of expected images corresponding to the expected cooking state of the first food ingredient according to the cooking operation of the cooking device are obtained, and Receiving user input to select a predicted image from among the plurality of predicted images above, and While the above cooking device performs a cooking operation, the camera is used to photograph the cooking chamber to obtain a second image, and A cooking device that transmits a notification regarding the first food ingredient to an external electronic device through the communication circuit when the cooking state of the first food ingredient within the second image is identified as corresponding to the expected image.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, A food ingredient image including the first food ingredient is obtained from the second image above, and Identifying the similarity between a first color vector corresponding to the above food ingredient image and a second color vector corresponding to the above predicted image, and A cooking device that identifies the cooking state of the first food ingredient within the second image as corresponding to the expected image when the similarity is identified as being greater than or equal to a threshold value.

3. In Paragraph 2, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, Calculate the average of color vectors for multiple pixels of the above food ingredient image to obtain a first color vector corresponding to the above food ingredient image, and A cooking device that calculates the average of color vectors for a plurality of pixels of the above-mentioned predicted image to obtain a second color vector corresponding to the above-mentioned predicted image.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, A food ingredient image including the first food ingredient is obtained from the first image above, and The above-mentioned food ingredient image, the type of the above-mentioned first food ingredient, and operation information set in the above-mentioned cooking device are input into an artificial intelligence model to obtain the above-mentioned plurality of predicted images from the artificial intelligence model, and The above operation information includes at least one of a cooking time, a cooking temperature, and a cooking mode, a cooking device.

5. In Paragraph 4, The above multiple expected images are, A cooking device comprising a plurality of expected images corresponding to the cooking state of the first food ingredient expected at a plurality of points in time within the cooking time as the cooking device performs the cooking operation based on the operation information.

6. In Paragraph 4, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, After the above cooking operation is stopped, receive user input to select the first food ingredient from at least one food ingredient in the cooking chamber, and A cooking device that, when the cooking device performs a cooking operation again, acquires a plurality of expected images of the first food ingredient based on the cooking state of the first food ingredient and the remaining cooking time.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, When a first food ingredient and a second food ingredient are selected based on user input among a plurality of food ingredients identified in the first image, the cooking state of the first food ingredient and the second food ingredient within the second image is identified as corresponding to the expected image for the first food ingredient and the expected image for the second food ingredient, respectively. If the cooking state of the first food ingredient within the second image is identified as corresponding to an expected image of the first food ingredient, a notification regarding the first food ingredient is transmitted to the external electronic device through the communication circuit, and A cooking device that transmits a notification regarding the second food ingredient to an external electronic device through the communication circuit when the cooking state of the second food ingredient within the second image is identified as corresponding to an expected image of the second food ingredient.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, Receiving user input to select a first predicted image and a second predicted image among a plurality of predicted images for the first food ingredient, and If the cooking state of the first food ingredient within the second image is identified as corresponding to the first expected image of the first food ingredient, a first notification regarding the first food ingredient is transmitted to the external electronic device through the communication circuit, and A cooking device that, when the cooking state of the first food ingredient within the second image is identified as corresponding to a second expected image of the first food ingredient, transmits a second notification regarding the first food ingredient to the external electronic device through the communication circuit.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the cooking device, Based on receiving user input for selecting all ingredients in the above cooking chamber, a plurality of expected images corresponding to the all ingredients expected according to the cooking operation of the above cooking device are obtained, and Receiving user input to select a predicted image from among the plurality of predicted images above, and A cooking device that transmits a notification regarding the first food ingredient to an external electronic device through the communication circuit when the cooking state of the entire food ingredient within the second image is identified as corresponding to the expected image.

10. A method for controlling a cooking device including a cooking chamber, A step of identifying at least one food ingredient in the cooking chamber based on a first image acquired through a camera included in the cooking device; When a first food ingredient is selected from the above at least one food ingredient based on user input, a step of acquiring a plurality of expected images corresponding to the expected cooking state of the first food ingredient according to the cooking operation of the cooking device; A step of receiving user input to select a predicted image from among the plurality of predicted images; A step of obtaining a second image by photographing the cooking chamber using the camera while the cooking device performs a cooking operation; and A control method comprising the step of transmitting a notification regarding the first food ingredient to an external electronic device when the cooking state of the first food ingredient within the second image is identified as corresponding to the expected image.

11. In Paragraph 10, The step of transmitting a notification regarding the first food ingredient to an external electronic device is: A step of obtaining an image of a food ingredient including the first food ingredient from the second image; A step of identifying the similarity between a first color vector corresponding to the above-mentioned food ingredient image and a second color vector corresponding to the above-mentioned predicted image; and A control method comprising the step of identifying that the cooking state of the first food ingredient in the second image corresponds to the expected image when the similarity is identified to be greater than or equal to a threshold value.

12. In Paragraph 11, The step of transmitting a notification regarding the first food ingredient to an external electronic device is: A step of calculating the average of color vectors for a plurality of pixels of the above-mentioned food ingredient image to obtain a first color vector corresponding to the above-mentioned food ingredient image; and A control method comprising the step of calculating the average of color vectors for a plurality of pixels of the above-mentioned predicted image to obtain a second color vector corresponding to the above-mentioned predicted image.

13. In Paragraph 10, The step of acquiring the above-mentioned plurality of expected images is, A step of obtaining a food ingredient image including the first food ingredient in the first image; and The method includes the step of inputting the above-mentioned food ingredient image, the above-mentioned type of the first food ingredient, and operation information set in the above-mentioned cooking device into an artificial intelligence model to obtain the above-mentioned plurality of predicted images from the artificial intelligence model; The above operation information is a control method including at least one of cooking time, cooking temperature, and cooking mode.

14. In Paragraph 13, The above multiple expected images are, A control method comprising a plurality of predicted images corresponding to the cooking state of the first food ingredient expected at a plurality of points in time within the cooking time, as the cooking device performs the cooking operation based on the operation information.

15. In Paragraph 13, The step of acquiring the above-mentioned plurality of expected images is, A step of receiving user input for selecting the first food ingredient from at least one food ingredient in the cooking chamber after the above cooking operation is stopped; and A control method comprising the step of acquiring a plurality of predicted images of the first food ingredient based on the cooking state of the first food ingredient and the remaining cooking time when the cooking device performs a cooking operation again.

Images