Robot, operation method thereof, and storage medium

Abstract

A robot is disclosed. A robot comprises a communication circuit, at least one sensor, at least one processor including processing circuitry, and a memory storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, instruct the robot to: identify feedback information corresponding to a cooking state of a cooking product on the basis of feature information of the cooking product identified on the basis of sensing data obtained through the at least one sensor and target feature information corresponding to the cooking product; and transmit a control command corresponding to the identified feedback information to an external cooking device through the communication circuit.

Description

Robot and its operation method and storage medium

[0001] The present disclosure relates to a robot, a method of operation thereof, and a storage medium, and more specifically, to a robot that performs a feedback operation for a food item, a method of operation thereof, and a storage medium.

[0002] Driven by advancements in electronic technology, various types of electronic devices are being developed and distributed, and recently, technological development for robots that provide services to users has been active. In particular, technological development for robots that provide various types of services related to the serving of food is becoming increasingly active.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] The aspects of the present disclosure are to address at least the problems and / or disadvantages mentioned above and to provide at least the advantages described below. Accordingly, the aspects of the present disclosure provide a robot that performs a feedback action on a food item and a method of operating the same.

[0005] Additional aspects will be presented in part in the following description, and in part may become apparent from the description or be learned by practicing the presented embodiments.

[0006] A robot according to one or more embodiments of the present disclosure comprises a communication circuit, at least one sensor, at least one processor including a processing circuitry, and a memory for storing instructions, wherein when the instructions are executed individually or collectively by the at least one processor, the robot identifies feedback information corresponding to the cooking state of the food based on characteristic information of the food identified based on sensing data acquired through the at least one sensor and target characteristic information corresponding to the food, and transmits a control command corresponding to the identified feedback information to an external cooking device through the communication circuit.

[0007] A method of operation of a robot according to one or more embodiments of the present disclosure includes an operation of identifying feedback information corresponding to the cooking state of a food item based on characteristic information of the food item identified based on sensing data acquired through at least one sensor and target characteristic information corresponding to the food item, and an operation of transmitting a control command corresponding to the identified feedback information to an external cooking device.

[0008] One or more non-transient computer-readable storage media storing one or more computer programs comprising computer-executable instructions that cause the robot to perform operations when executed individually or collectively by at least one processor of the robot, wherein the operations identify feedback information corresponding to the cooking state of the food based on characteristic information of the food identified based on sensing data acquired through at least one sensor and target characteristic information corresponding to the food, and cause to transmit a control command corresponding to the identified feedback information to an external cooking device.

[0009] The above and other aspect features and advantages of specific embodiments of the present disclosure will become more apparent from the following description, which is referenced together with the accompanying drawings.

[0010] FIG. 1 is a diagram for schematically illustrating a robot according to one embodiment.

[0011] FIG. 2 is a block diagram showing the configuration of a robot according to one embodiment.

[0012] FIG. 3 is a flowchart illustrating a method of operation of a robot according to one embodiment.

[0013] FIG. 4 is a flowchart illustrating a method for performing a feedback operation according to one embodiment.

[0014] FIG. 5 is a flowchart illustrating a method for identifying feedback information corresponding to temperature according to one embodiment.

[0015] FIG. 6 is a flowchart illustrating a method for identifying feedback information corresponding to a scent according to one embodiment.

[0016] FIG. 7a is a flowchart illustrating a method for providing cooking evaluation information according to one embodiment.

[0017] FIGS. 7b and FIGS. 7c are drawings for explaining a method of providing cooking evaluation information according to one embodiment.

[0018] FIGS. 8A, FIGS. 8B, and FIGS. 8C are drawings for illustrating a robot according to one embodiment.

[0019] FIG. 9 is a diagram illustrating a method for performing a feedback operation according to one embodiment.

[0020] FIG. 10 is a drawing for explaining a method for identifying plating information according to one embodiment.

[0021] FIG. 11 is a drawing for explaining a cleaning operation according to one embodiment.

[0022] FIG. 12 is a block diagram showing the detailed configuration of a robot according to one embodiment.

[0023] It should be noted that the same reference numbers are used throughout the drawings to denote identical or similar elements, features, and structures.

[0024] The following description, made with reference to the attached drawings, is provided to facilitate a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. While various specific details are included to aid understanding, they should be considered merely illustrative. Accordingly, those skilled in the art will recognize that various changes and modifications to the various embodiments described herein may be made without departing from the scope and spirit of the present disclosure. Additionally, for the sake of clarity and brevity, descriptions of well-known functions and configurations may be omitted.

[0025] The terms and words used in the following description and claims are not limited to their bibliographic meanings and are used merely to enable the inventor to understand the present disclosure clearly and consistently. Accordingly, it is evident to those skilled in the art that the following description of various embodiments of the present disclosure is provided merely for illustrative purposes and is not intended to limit the present disclosure as defined by the appended claims and their equivalents.

[0026] It should be understood that, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" include plural referents. Thus, for example, a reference to "a component surface" includes a reference to one or more such surfaces.

[0027] Terms used in the specification will be briefly explained, and then the present disclosure will be explained.

[0028] Commonly used terms currently in use have been selected for use in the embodiments of this disclosure in consideration of their functions within this disclosure, and these terms may be changed according to the intent of those skilled in the art, case law, the emergence of new technology, etc. Additionally, in specific cases, terms may be selected at the applicant's discretion. In such cases, the meaning of such terms is mentioned in the relevant explanatory section of this disclosure. Accordingly, terms used in this disclosure should be defined based on their meaning and the content throughout this disclosure, rather than merely their names.

[0029] In the specification, expressions such as "have," "may have," "include," or "may include" indicate the presence of the relevant feature (e.g., a component such as a numerical value, function, behavior, or part) and do not exclude the presence of additional features.

[0030] Expressions such as "at least one of A or / and B" can represent "A or B" or "both A and B".

[0031] Expressions such as "first" and "second" used in this disclosure may refer to various components regardless of order or importance. These expressions are used solely to distinguish one component from another and do not limit said component.

[0032] Where a component (e.g., Component 1) is referred to as being "(functionally or telecommunicationally) coupled / coupled" or "connected" to another component (e.g., Component 2), it should be understood that the component may be directly coupled to the other component or coupled through another component (e.g., Component 3).

[0033] A singular expression may include a plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0034] In the present disclosure, a "module" or "a component / or" may perform at least one function or operation and may be implemented in hardware, software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "components / or" may be integrated into at least one module and implemented by a processor (not shown), except for a "module" or "component / or" that must be implemented in specific hardware.

[0035] Furthermore, terms such as "signal" in the specification may include not only electrical signals but also signals in the form of sound waves, and electrical signals may be analog signals or digital signals. For example, expressions such as "audio signal (or noise signal)" indicate a sound wave (or radio wave) signal when the signal is outside the electronic device, and an electrical signal when it is inside the electronic device depending on its location. Additionally, signal processing inside the electronic device described below may be digital signal processing, analog signal processing, or a signal processing method that combines analog and digital methods.

[0036] Additionally, terms such as "filter" in the specification refer to a device for removing specific components (e.g., a specific frequency range or a specific pattern), and the filter may be a digital filter or an analog filter.

[0037] It should be understood that the blocks of each flowchart and combinations of flowcharts can be executed by one or more computer programs containing computer-executable instructions. One or more computer programs may be stored entirely in a single memory device, or one or more computer programs may be divided into different parts stored in several different memory devices.

[0038] Any of the functions or operations described herein may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing and includes circuits such as an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural network processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, a Bluetooth™ chip, a Global Positioning System (GPS) chip, a Near Field Communication (NFC) chip, a connectivity chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio codec (CODEC) chip, a Universal Serial Bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system-on-chip (SoC), an IC, etc.

[0039] FIG. 1 is a diagram for schematically illustrating a robot according to one embodiment.

[0040] Referring to FIG. 1, a robot (100) according to one embodiment can identify characteristic information of food items (1, 2, 3, 4, 5 and 6). According to one example, the feature information may include at least one of the following: information related to the state of the food (1, 2, 3, 4, 5 and 6), for example, temperature information of the food (1, 2, 3, 4, 5 and 6), salinity information of the food (1, 2, 3, 4, 5 and 6), capsaicin concentration information of the food (1, 2, 3, 4, 5 and 6), appearance (color or plating) information of the food (1, 2, 3, 4, 5 and 6), texture information of the food (1, 2, 3, 4, 5 and 6), or flavor information of the food (1, 2, 3, 4, 5 and 6). According to one example, the robot (100) may identify the feature information using sensing data obtained through a sensor.

[0041] A robot (100) according to one embodiment can identify feedback information of the cooked food (1, 2, 3, 4, 5, and 6). According to one example, the robot (100) can identify feedback information corresponding to the cooking state of the cooked food (1, 2, 3, 4, 5, and 6) based on feature information and target feature information corresponding to the cooked food (1, 2, 3, 4, 5, and 6). According to one example, the target feature information of the cooked food (1, 2, 3, 4, 5, and 6) may be information corresponding to the state that the cooked food (1, 2, 3, 4, 5, and 6) should ideally reach.

[0042] A robot (100) according to one embodiment can perform an action corresponding to identified feedback information. According to one example, if the robot (100) identifies feedback information that 'the temperature of the food (1, 2, 3, 4, 5 and 6) needs to be raised by 5 degrees,' the robot (100) can transmit a control command related thereto to an external cooking device or the robot (100) can perform the action directly.

[0043] FIG. 2 is a block diagram showing the configuration of a robot according to one embodiment.

[0044] According to FIG. 2, the robot (100) may include a communication circuit (110), at least one sensor (120), at least one processor (130) and a memory (140).

[0045] The robot (100) may be implemented as a different type of device that travels through a travel space. In one example, the robot (100) may be a robot that moves to a specific location and provides a service to a user. For example, the robot (100) may travel through a travel space and perform a service related to food. In one example, the robot (100) may be a different type of travel robot including a wheel robot, but is not limited thereto. The robot (100) of the present invention will be described in detail through FIGS. 8a to 8c.

[0046] The communication circuit (110) can input and output various types of data. For example, the communication circuit (110) can transmit and receive various types of data to and from an external device (e.g., source device), an external storage medium (e.g., USB memory), an external server (e.g., web hard drive) through communication methods such as AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), Optical, Coaxial, etc.

[0047] According to one example, the communication circuit (110) may include a BLE (Bluetooth Low Energy) module. BLE refers to Bluetooth technology capable of transmitting and receiving low-power, low-capacity data in a 2.4 GHz frequency band with a range of about 10 m. However, it is not limited thereto, and the communication circuit (110) may include a Wi-Fi communication module. That is, the communication circuit (110) may include at least one of a BLE (Bluetooth Low Energy) module or a Wi-Fi communication module.

[0048] At least one sensor (120, hereinafter referred to as the sensor) may include a plurality of sensors of various types. The sensor (120) may measure physical quantities or detect the operating state of the robot (100) and convert the measured or detected information into an electrical signal. The sensor (120) may include a camera (or a camera sensor), and the camera may include a lens that focuses visible light or other optical signals received by being reflected by an object onto an image sensor, and an image sensor capable of detecting visible light or other optical signals. Here, the image sensor may include a 2D pixel array divided into a plurality of pixels. Alternatively, at least one sensor (120) may include a temperature sensor or an infrared sensor. Alternatively, according to one example, at least one sensor (120) may include at least one of a salinity sensor that measures the salinity of a food, an olfactory sensor that senses the aroma of a food, and a texture sensor. Alternatively, according to one example, at least one sensor (120) may include a sensor that measures the pH (potential of hydrogen) of a food. This will be described later.

[0049] At least one processor (130) (hereinafter, processor) is electrically connected to a communication circuit (110), at least one sensor (120), and a memory (140) to control the overall operation of the robot (100). The processor (130) may be composed of one or more processors. Specifically, the processor (130) may perform the operation of the robot (100) according to various embodiments of the present disclosure by executing at least one instruction stored in the memory (140).

[0050] According to one embodiment, the processor (130) may be implemented as a digital signal processor (DSP) that processes digital video signals, a microprocessor, a Graphics Processing Unit (GPU), an Artificial Intelligence (AI) processor, a Neural Processing Unit (NPU), or a Time Controller (TCON). However, it is not limited thereto, and may include or be defined by one or more of a central processing unit (CPU), a Micro Controller Unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor. Additionally, the processor (130) may be implemented as a System on Chip (SoC) or Large Scale Integration (LSI) with a built-in processing algorithm, or may be implemented in the form of an Application Specific Integrated Circuit (ASIC) or Field Programmable Gate Array (FPGA).

[0051] The memory (140) can store data necessary for various embodiments. Depending on the purpose of data storage, the memory (140) may be implemented in the form of a memory embedded in the robot (100) or in the form of a memory that can be attached to and detached from the robot (100). For example, data for driving the robot (100) may be stored in a memory embedded in the robot (100), and data for the expansion function of the robot (100) may be stored in a memory that can be attached to and detached from the robot (100).

[0052] Meanwhile, the memory embedded in the robot (100) may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD). Additionally, the memory that can be attached to the robot (100) may be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.) or external memory that can be connected to a USB port (e.g., USB memory).

[0053] According to one embodiment, the processor (130) may acquire sensing data through at least one sensor (120). According to one example, at least one sensor (120) may include a camera sensor. Alternatively, at least one sensor (120) may include a temperature sensor. Alternatively, at least one sensor (120) may include an olfactory sensor. According to one example, the olfactory sensor may be a sensor that detects odors and quantitatively measures their type and concentration. According to one example, the texture sensor may be a sensor that detects surface characteristics of an object and acquires tactile information such as roughness, hardness, and elasticity. According to one example, the processor (130) may acquire texture information of the food based on the sensing data acquired through the texture sensor.

[0054] According to one example, the olfactory sensor may be a sensor that measures the type and intensity of a scent by detecting and identifying specific scent molecules in the air. According to one example, the processor (130) may acquire sensing data to detect the state of the food through at least one sensor (120).

[0055] According to one embodiment, the processor (130) can identify characteristic information of the food based on sensing data obtained through at least one sensor (120). According to one example, the characteristic information of the food is information related to the state of the food and may include, for example, at least one of temperature information of the food, salt content information of the food, capsaicin concentration information of the food, appearance (color or plating) information of the food, texture information of the food, or aroma information of the food.

[0056] According to one example, the processor (130) can obtain plating information of the food through a camera sensor. According to one example, the plating information of the food may be image information regarding the plating state of the food, but is not limited thereto.

[0057] According to one embodiment, the processor (130) may obtain target feature information corresponding to the food item. According to one example, the target feature information of the food item may be information corresponding to the state that the food item should ideally reach. For example, it may include the appropriate temperature of the food item, the ideal color of the food item, and the ideal texture of the food item, but is not limited thereto. Meanwhile, the processor (130) may obtain target feature information corresponding to each type of food item (e.g., type of food) from an external server, but is not limited thereto, and according to one example, the target feature information corresponding to each type of food item may be stored in memory (140).

[0058] According to one embodiment, the processor (130) can compare characteristic information of the food with target characteristic information corresponding to the food. According to one example, the processor (130) can compare the characteristic information of the identified food with target characteristic information corresponding to the food. For example, the processor (130) can evaluate the plating state of the food by comparing the plating information of the food with the target characteristic information of the food.

[0059] According to one embodiment, the processor (130) can identify feedback information corresponding to the cooking state of the food based on a comparison result. According to one example, the feedback information is information indicating the difference between the current cooking state of the food and the target state, and may be, for example, information such as 'the plating of the food needs to be improved,' 'the temperature of the food needs to be raised by 5 degrees,' or 'additional cooking of the food is needed,' but is not limited thereto.

[0060] According to one example, the processor (130) can identify feedback information corresponding to each type of feature information (e.g., appearance, temperature, texture, etc.).

[0061] According to one embodiment, the processor (130) can identify a control command corresponding to feedback information. According to one example, the control command may be a command that instructs an external cooking device to perform an action corresponding to the identified feedback information, but is not limited thereto. For example, it may be assumed that the identified feedback information is 'the temperature of the food needs to be raised by 5 degrees (°C)'. The processor (130) can identify a control command corresponding to a cooking device that manages the temperature of the food (e.g., a mass cooking device or a microwave oven, etc.) as a control command to raise the temperature of the food by 5 degrees.

[0062] According to one embodiment, the processor (130) can transmit a control command corresponding to the identified feedback information to an external cooking device through the communication circuit (110). According to one embodiment, the external cooking device refers to a device that performs actual cooking and may include, for example, an oven, an induction cooker, and a microwave oven, but is not limited thereto. According to one embodiment, the external cooking device may be a different type of device including a mass cooking device, a robotic fryer, an automatic stir-frying device, and a robot that performs direct fire operation on the food.

[0063] Alternatively, according to one embodiment, the processor (130) may perform a direct feedback action based on identified feedback information. According to one example, if the identified feedback information is that the temperature of the food needs to be raised by 5 degrees (°C), the processor (130) may move to a mass cooking device and raise the cooking temperature of the cooking device. Alternatively, according to one example, if the identified feedback information is that the plating of the food needs to be improved, the processor (130) may drive a robot hand to modify the plating of the food. This will be described later.

[0064] According to one embodiment, the processor (130) may obtain user feedback information regarding the food. The user feedback information regarding the food may be information such as "the salt content of the first food is too high" or "the temperature of the second food is too low," but is not limited thereto. According to one example, the processor (130) may obtain user feedback information from an external server, but is not limited thereto. According to one example, if the robot includes a user interface, the processor (130) may receive user feedback information through the user interface.

[0065] According to one example, when the processor (130) receives user feedback information, it can transmit a control command corresponding to the received feedback information to an external cooking device or directly perform an operation corresponding to the feedback information.

[0066] Alternatively, according to one example, the processor (130) may update the target feature information based on user feedback information. For example, if user feedback information corresponding to "the salt content of the first dish is too high" is received, the processor (130) may lower the target salt content of the first dish compared to before. Alternatively, for example, the processor (130) may modify the recipe information corresponding to the first dish. By adjusting the target feature information to reflect user feedback, the robot is able to produce a cooking result that is more suited to the user's taste.

[0067] According to the example described above, the robot can monitor the cooking process and the food through at least one sensor (120), provide appropriate feedback, and control an external cooking device to support obtaining optimal cooking results. In addition, it can learn user feedback to continuously improve the cooking process.

[0068] FIG. 3 is a flowchart for explaining the operation method of a robot (100) according to one embodiment.

[0069] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0070] Referring to FIG. 3, according to one embodiment, the operation method may include an operation (S310) of identifying feedback information corresponding to the cooking state of a food item based on characteristic information of the food item identified based on sensing data obtained through at least one sensor (120) and target characteristic information corresponding to the food item.

[0071] According to one example, the robot (100) can acquire sensing data through at least one sensor (120). According to one example, the robot (100) can identify characteristic information of a food item based on the acquired sensing data. According to one example, the robot (100) can identify feedback information by comparing the identified characteristic information of the food item with target characteristic information corresponding to the food item.

[0072] According to one embodiment, the operation method may include an operation (S320) of transmitting a control command corresponding to identified feedback information to an external cooking device through a communication circuit (110).

[0073] According to one example, when feedback information is identified, the robot (100) may transmit a control command corresponding to the identified feedback information to an external cooking device through a communication circuit (110). According to one example, the control command may be a command that instructs the external cooking device to perform an action corresponding to the identified feedback information, but is not limited thereto. Alternatively, according to one example, the robot (100) may perform a direct feedback action based on the identified feedback information.

[0074] FIG. 4 is a flowchart illustrating a method for performing a feedback operation according to one embodiment.

[0075] Referring to FIG. 4, according to one embodiment, the operation method may include an operation (S410) of identifying plating information of a food item based on sensing data obtained through a camera sensor.

[0076] According to one example, at least one sensor (120) may include a camera sensor. According to one example, the feature information may include plating information of the food. According to one example, the robot (100) may identify plating information corresponding to the food through the camera sensor. According to one example, the plating information of the food may be image information regarding the plating state of the food, but is not limited thereto. Alternatively, according to one example, the plating information may include information regarding the ratio between the main ingredient and the auxiliary ingredient, and information regarding the amount of solid ingredients.

[0077] According to one embodiment, the operation method may include an operation (S420) of performing a feedback operation on a food item based on feedback information when feedback information corresponding to the plating is identified by comparing identified plating information and target feature information corresponding to the food item.

[0078] According to one example, the robot (100) can identify the type of food based on sensing data acquired through a camera sensor and identify target feature information corresponding to the identified type of food. For example, the robot (100) can identify the type of food by inputting the acquired sensing data into a learned neural network model and identify target feature information corresponding to the identified type based on information stored in memory (140). According to one example, the learned neural network model may be a model trained to classify the type of object (e.g., food) included in an image when an image is input.

[0079] According to one example, the robot (100) can compare identified plating information and target feature information corresponding to the food. According to one example, the robot (100) can compare plating information (or an image corresponding to plating information) and target feature information corresponding to the food. According to one example, the target feature information corresponding to the food may include an appearance image corresponding to the food. The robot (100) can identify feedback information by comparing the plating information and the appearance image corresponding to the target feature information. For example, the feedback information related to the current plating state of the food may be of different types, including 'excessive decoration', 'color mismatch', and 'difficult-to-eat structure', but is not limited thereto.

[0080] According to one example, the robot (100) may perform a feedback action on the food based on feedback information. For example, the robot (100) may perform an action of removing some of the decorations present on the top of the food using the robot (100) hand. Or, for example, the robot (100) may perform a plating change action using the robot (100) hand to make the colors of the food harmonious. Or, for example, the robot (100) may perform a plating change action to make the food easy to eat.

[0081] According to one example, the robot (100) may obtain feedback information using a learned neural network model. According to one example, the neural network model may be a model trained to output feedback information corresponding to the plating when plating information and target feature information are input. According to one example, the neural network model may be a Convolutional Neural Network (CNN) model and may be a model trained to output information about the plating state of the food. For example, the feedback information may be different types of information including 'excessive decoration', 'color mismatch', and 'difficult-to-eat structure'. However, it is not limited thereto.

[0082] FIG. 5 is a flowchart illustrating a method for identifying feedback information corresponding to temperature according to one embodiment.

[0083] Referring to FIG. 5, according to one embodiment, the operation method may include an operation (S510) of identifying temperature information of a food item based on sensing data obtained through a temperature sensor.

[0084] According to one example, at least one sensor (120) may include a temperature sensor. According to one example, the feature information may include temperature information of the food. According to one example, the robot (100) may obtain temperature information of the food using the temperature sensor.

[0085] According to one embodiment, the operation method may include an operation (S520) of comparing identified temperature information and target feature information corresponding to the food being cooked, and when feedback information corresponding to the temperature is identified, transmitting a control command corresponding to the feedback information to an external cooking device through a communication circuit (110).

[0086] According to one example, when temperature information of a food item is identified, the robot (100) can compare the identified temperature information with target feature information corresponding to the food item. According to one example, the target feature information corresponding to the food item may include target temperature information corresponding to the food item. According to one example, the robot (100) can identify feedback information by comparing the temperature information and the target feature information. For example, the feedback information may be 'the temperature of the food item needs to be raised by 3 degrees,' but is not limited thereto.

[0087] According to one example, when feedback information corresponding to the temperature of the food is identified, the robot (100) can transmit a control command corresponding to the feedback information to an external cooking device through a communication circuit (110). For example, the robot (100) can identify a control command corresponding to a cooking device that manages the temperature of the food (e.g., a mass cooking device or a microwave oven, etc.) as a control command to raise the temperature of the food by 3 degrees. Alternatively, for example, the robot (100) may directly perform an operation to control a cooking device that manages the temperature of the food.

[0088] According to one example, the robot (100) can obtain feedback information using a learned neural network model. According to one example, the neural network model may be a model trained to output feedback information corresponding to the temperature when temperature information and target feature information are input. According to one example, the neural network model may be a Long Short-Term Memory (LSTM) model and may be a model trained to output feedback information regarding the temperature state of the food being cooked. According to one example, the robot (100) can obtain feedback information corresponding to the temperature based on output data. According to one example, the LSTM model may output feedback information corresponding to the temperature of the food being cooked based on information regarding a sequence of temperatures based on a time series.

[0089] FIG. 6 is a flowchart illustrating a method for identifying feedback information corresponding to a scent according to one embodiment.

[0090] Referring to FIG. 6, according to one embodiment, the operation method may include an operation (S610) of identifying aroma information of a food based on sensing data obtained through an olfactory sensor.

[0091] According to one example, at least one sensor (120) may include an olfactory sensor. According to one example, the feature information may include aroma information of the food. According to one example, the robot (100) may obtain aroma information of the food using the olfactory sensor.

[0092] According to one embodiment, the operation method may include an operation (S620) of transmitting a control command corresponding to the feedback information to an external cooking device through a communication circuit (110) when feedback information corresponding to the scent is identified by comparing the identified scent information and target feature information corresponding to the cooking material.

[0093] According to one example, when aroma information of a food item is identified, the robot (100) can compare the identified aroma information with target feature information corresponding to the food item. According to one example, the target feature information corresponding to the food item may include target aroma information corresponding to the food item. According to one example, the robot (100) can identify feedback information by comparing the aroma information and the target feature information. For example, the feedback information may be "the aroma of the spices in the food item is too strong," but is not limited thereto.

[0094] According to one example, when feedback information corresponding to the aroma of the food is identified, the robot (100) can transmit a control command corresponding to the feedback information to an external cooking device through a communication circuit (110). For example, the robot (100) can identify a control command to reduce the aroma of a specific spice included in the first food, which causes a cooking device cooking the first food to reduce the amount of a specific spice used.

[0095] According to one example, the robot (100) can obtain feedback information using a learned neural network model. According to one example, the neural network model may be a model trained to output feedback information corresponding to the scent when scent information and target feature information are input. According to one example, the neural network model may be a graph neural network (GNN) model and may be a model trained to output feedback information regarding the scent state of the food. According to one example, the robot (100) can obtain feedback information corresponding to the scent based on output data.

[0096] FIG. 7a is a flowchart illustrating a method for providing cooking evaluation information according to one embodiment. FIG. 7b and FIG. 7c are drawings illustrating a method for providing cooking evaluation information according to one embodiment.

[0097] Referring to FIG. 7a, according to one embodiment, the operation method may include an operation (S710) of identifying characteristic information of the updated food when the cooking state of the food is updated based on feedback information.

[0098] According to one example, when the robot (100) receives feedback information from a user, it can transmit a control command corresponding to the received feedback information to an external cooking device or perform an action corresponding to the feedback information.

[0099] According to one example, the robot (100) can update the cooking status of the food after a control command corresponding to the feedback information is transmitted to an external cooking device or after performing an action corresponding to the feedback information. For example, the robot (100) can obtain characteristic information of the updated food based on sensing data obtained through at least one sensor (120).

[0100] According to one embodiment, the operation method may include an operation (S720) of providing cooking evaluation information obtained based on updated cooking feature information.

[0101] According to one example, the robot (100) can obtain cooking evaluation information based on updated cooking feature information and target feature information corresponding to the cooking. According to one example, the cooking evaluation information may include a value obtained by comparing the state of the cooking with the target state.

[0102] According to one example, referring to FIG. 7b, the robot (100) may acquire evaluation information corresponding to each state of the food and provide a UI (710) containing the evaluation information. For example, the evaluation information may include salt content information of the food, capsaicin concentration information (or spicy taste information), pH (potential of hydrogen) concentration, information on the ratio between main ingredients and auxiliary ingredients, solid content information, temperature information at the time of serving the food, temperature information by time of serving the food, aroma information of the food, and texture information of the food. According to one example, the robot (100) may acquire evaluation information corresponding to each of different types of food and provide a UI (710) containing evaluation information corresponding to each of the food.

[0103] Alternatively, according to one example, referring to FIG. 7c, the robot (100) may provide a UI (720) containing information on relative scores corresponding to each evaluation item of the food. According to one example, the evaluation items of the food may include evaluation values ​​for at least one of the taste (e.g., saltiness or capsaicin concentration), texture, appearance, temperature, and aroma of the food. According to one example, the evaluation values ​​may be measured based on similarity with target feature information, but are not limited thereto.

[0104] FIGS. 8a to 8c are drawings for explaining a robot (100) according to one embodiment.

[0105] Referring to FIG. 8a, according to one embodiment, the robot (100) may be implemented as a robot (100) including an end effector (810).

[0106] According to one example, the robot (100) may include a battery. According to one example, the robot (100) may include at least one manipulator. According to one example, the manipulator may be a robot (100) arm composed of seven axes and may perform the action of grasping an object by specifying the coordinates of the object in three-dimensional space. For example, the robot (100) may be implemented as a dual-arm robot.

[0107] According to one example, the robot (100) may include an end effector (810). According to one example, the end effector (810) may be made of rubber and may perform the action of picking up different types of objects. According to one example, the end effector (810) may be implemented as the hand of the robot (100). Alternatively, the end effector (810) may include different types of sensors, including a salinity sensor, a temperature sensor, and a texture sensor. This will be explained in detail through FIGS. 8b and 8c.

[0108] According to one example, the robot (100) may include a drive unit. According to one example, the drive unit may include a plurality of wheels, and the robot (100) may control the drive unit to perform driving motions. The drive unit is a drive system that enables the robot (100) to move freely in all directions, and enables the robot (100) to perform forward, backward, left, right, and diagonal movements.

[0109] According to one example, the robot (100) may include a depth camera. According to one example, the robot (100) may recognize objects present around the robot (100) using the depth camera. According to one example, the robot (100) may perform an eye lighting action. According to one example, the robot (100) may include a light-emitting element and may perform an emotion expression action of the robot (100) using the light-emitting element. For example, the robot (100) may perform a smiling action, a blank expression, or a blinking action using the light-emitting element.

[0110] According to one example, the robot (100) may include a rear light-emitting element that indicates the driving direction of the robot (100). For example, an operation in which an LED light is continuously turned on in the driving direction of the robot (100) may be performed. According to one example, the robot (100) may include a Lidar sensor. According to one example, the robot (100) may include a waterproof function corresponding to a joint including a wrist.

[0111] According to one example, motion information corresponding to multiple emotional expressions may be stored in the memory (140). According to one example, an emotional expression refers to various emotional states that the robot (100) can express, and motion information refers to data representing the physical or visual motion of the robot (100) corresponding to each emotional expression.

[0112] According to one example, the robot (100) can identify one of a plurality of emotional expressions based on cooking evaluation information. According to one example, the cooking evaluation information may be evaluation data regarding the result or process of a cooking operation performed by the robot (100). According to one example, the robot (100) can identify motion information corresponding to the identified emotional expression based on information stored in memory.

[0113] According to one example, the robot (100) can perform a motion corresponding to identified motion information. For example, the robot (100) can perform a smiling motion, a blank expression, or a blinking motion using a light-emitting element. Or, for example, the robot (100) can output a voice corresponding to "The food is too delicious" through a speaker. Or, it can output a voice corresponding to "The food is too salty" through a speaker.

[0114] Referring to FIG. 8b and FIG. 8c, according to one example, the robot (100) may include an end effector (810). According to one example, the end effector (810) may be a device mounted on the end of the robot (100) arm to perform an action. According to one example, the end effector (810) may be implemented as a hand type of the robot (100). According to one example, the end effector (810) may be implemented in a shape similar to a human finger, and each finger portion may include a joint structure. According to one example, at least one sensor (120) may be provided in a part (820) of the end effector (810). For example, a different type of sensor, including a salinity sensor, a temperature sensor, and a texture sensor, may be provided in a part (820) of the end effector (810).

[0115] FIG. 9 is a diagram illustrating a method for performing a feedback operation according to one embodiment.

[0116] Referring to FIG. 9, according to one embodiment, the robot (100) can perform a feedback operation. According to one example, when feedback information corresponding to the cooking state of the food (90) is identified, the robot (100) can transmit a control command corresponding to the identified feedback information to an external cooking device (910 or 920) through a communication circuit (110). For example, the robot (100) may be in a state of communication connection with a mass cooking device (910), and the robot (100) can transmit a control command corresponding to the feedback information to the mass cooking device (910) through the communication circuit (110). The mass cooking device (910) that receives the control command can cook the food based on the control command.

[0117] Alternatively, according to one example, the robot (100) may be in a state of communication connection with an external device (920) that performs a transport operation of the food (90), and the robot (100) may transmit control commands to the external device (920). However, it is not limited thereto, and the robot (100) may control other types of cooking devices (e.g., an automatic stir-frying device or a robot (100) fryer, etc.) in addition to the external cooking device (910 or 920) shown in FIG. 9.

[0118] According to one example, the robot (100) can perform a cooking operation. For example, if the feedback information is 'the plating of the food needs improvement,' the robot (100) may drive the robot's (100) hand to modify the plating of the food (90). Alternatively, for example, if the feedback information is 'the salt content of the first food is too high,' the robot (100) may move to a cooking device that cooks the first food and directly perform an operation to lower the salt content of the first food.

[0119] FIG. 10 is a drawing for explaining a method for identifying plating information according to one embodiment.

[0120] Referring to FIG. 10, according to one embodiment, a robot (100) can acquire plating information (1000) of a food item based on sensing data acquired through at least one sensor (120). According to one example, at least one sensor (120) may include a camera sensor or a vision sensor, and the plating information (1000) may be image-type data. However, it is not limited thereto, and at least one sensor (120) may include a sensor other than the sensor described above.

[0121] According to one example, the robot (100) can obtain feedback information by comparing plating information (1000) of a dish with target feature information corresponding to the dish. According to one example, the robot (100) can obtain feedback information by inputting plating information (1000) and target feature information into a learned neural network model. According to one example, the feedback information may be different information including 'the amount of the first dish is too large' and 'the plating of the second dish needs to be improved,' but is not limited thereto.

[0122] FIG. 11 is a drawing for explaining a cleaning operation according to one embodiment.

[0123] Referring to FIG. 11, according to one embodiment, the robot (100) can perform a cleaning operation. According to one example, the robot (100) may further include a driving unit and can control the driving unit to perform a cleaning operation for the robot (100). For example, the robot (100) can move to a cleaning unit (1100) and perform a cleaning operation.

[0124] In one example, the robot (100) can perform a cleaning operation on the end effector. Alternatively, in one example, if the end effector is implemented as a robot (100) hand, the robot (100) can perform a cleaning operation on the robot (100) hand.

[0125] According to one example, the robot (100) may perform a washing operation after performing a feature information acquisition operation. For example, the robot (100) may touch the food through an end effector to measure the salt content of the food. Alternatively, the robot (100) may perform a plating operation for the food through an end effector. The robot (100) may perform a washing operation after the feature information acquisition operation for the first food is performed. The robot (100) may perform a feature information acquisition operation for the second food after performing the washing operation.

[0126] FIG. 12 is a block diagram showing the detailed configuration of a robot according to one embodiment.

[0127] According to FIG. 12, the robot (100') may include a communication circuit (110), at least one sensor (120), at least one processor (130), memory (140), display (150), user interface (160), speaker (170), microphone (180), and driving unit (190). A detailed description of configurations shown in FIG. 12 that overlap with configurations shown in FIG. 2 will be omitted.

[0128] The display (150) may be implemented as a display including a self-emissive element or as a display including a non-emissive element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, or a QLED (Quantum dot light-emitting diodes). The display (150) may also include a driving circuit, a backlight unit, etc., which can be implemented in the form of an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (150) may be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc. The processor (130) can control the display (150) to output an output image obtained according to the various embodiments described above. Here, the output image may be a high-resolution image of 4K or 8K or higher. According to one embodiment, the output image may be a game image.

[0129] According to one embodiment, the display (150) may include a plurality of haptic elements. The haptic elements may be implemented as motors to provide haptic feedback (e.g., vibration feedback) to a user, but are not limited thereto. According to one example, the display (150) may include a predetermined number of haptic elements. For example, the display (150) may include a predetermined number of haptic elements corresponding to a predetermined number of sub-regions of the display, but is not limited thereto, and it is obvious that the display may include a number of haptic elements different from the number of sub-regions corresponding to the display.

[0130] The user interface (160) is configured for the robot (100') to perform interaction with the user. For example, the user interface (160) may include at least one of a touch sensor, a motion sensor, a button, a jog dial, a switch, a microphone, or a speaker, but is not limited thereto.

[0131] According to one embodiment, the speaker (170) may be composed of a tweeter for reproducing high-frequency sound, a midrange for reproducing mid-frequency sound, a woofer for reproducing low-frequency sound, a subwoofer for reproducing ultra-low-frequency sound, an enclosure for controlling resonance, and a crossover network for dividing the frequency of an electrical signal input to the speaker into bands.

[0132] According to one embodiment, the speaker (170) can output an acoustic signal to the outside of the robot (100'). The speaker (170) can output multimedia playback, recording playback, various notification sounds, voice messages, etc. The robot (100') may include an audio output device such as the speaker (170), but may include an output device such as an audio output terminal. In particular, the speaker (170) can provide acquired information, information processed or produced based on the acquired information, response results to user voice, or operation results, etc., in the form of voice.

[0133] The microphone (180) may refer to a module that acquires sound and converts it into an electrical signal, and may be a condenser microphone, ribbon microphone, moving coil microphone, piezoelectric element microphone, carbon microphone, or MEMS (Micro Electro Mechanical System) microphone. Additionally, it may be implemented in omnidirectional, bidirectional, unidirectional, subcardioid, supercardioid, or hypercardioid modes. According to one embodiment, the robot (100') may include the microphone (180) and an inner microphone, and the microphone (180) may be a microphone located relatively outside the body. According to one example, the robot (100') may acquire an audio signal including external noise through the microphone (180). According to one embodiment, the microphone (180) may be positioned in a direction opposite to the direction in which the speaker (170) emits sound.

[0134] The drive unit (190) is a device capable of driving the robot (100'). The drive unit (190) can control the driving direction and driving speed according to the control of the processor (130). In one example, the drive unit (190) may include a power generation device that generates power for the robot (100') to drive (e.g., a gasoline engine, a diesel engine, an LPG (liquefied petroleum gas) engine, an electric motor, etc., depending on the fuel (or energy source) used), a steering device for controlling the driving direction (e.g., manual steering, hydraulics steering, electronic control power steering (EPS), etc.), and a driving device that drives the robot (100') according to the power (e.g., wheels, propellers, etc.). Here, the drive unit (190) may be modified according to the driving type of the robot (100') (e.g., wheel type, walking type, flying type, etc.).

[0135] According to the example described above, the robot (100') can monitor the cooking process and the food through at least one sensor (120), provide appropriate feedback, and control an external cooking device to support obtaining optimal cooking results. In addition, it can learn user feedback to continuously improve the cooking process.

[0136] Meanwhile, according to the exemplary embodiments of the present disclosure, the various embodiments described above may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include a display device (e.g., a display device (A)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code provided or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0137] Additionally, according to one embodiment, the method according to the various embodiments described above may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed online in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or provided on a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0138] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the functions performed by each of the respective components prior to integration in the same or similar manner. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0139] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In robots, Communication circuit; At least one sensor; At least one processor including processing circuitry; and Includes memory for storing instructions; and When the above instructions are executed individually or collectively by the at least one processor, the robot, Based on the characteristic information of the food identified based on the sensing data acquired through the at least one sensor and the target characteristic information corresponding to the food, feedback information corresponding to the cooking state of the food is identified, and A robot that transmits a control command corresponding to the above-mentioned identified feedback information to an external cooking device through the communication circuit.

2. In Paragraph 1, The above-mentioned at least one sensor is, Including a camera sensor; further The above feature information is, It includes plating information for the above-mentioned dish, When the above instructions are executed individually or collectively by the at least one processor, the robot, Identifying plating information of the food based on sensing data acquired through the camera sensor, and A robot that performs a feedback operation on the food based on the feedback information when feedback information corresponding to the plating is identified by comparing the identified plating information and the target feature information corresponding to the food.

3. In Paragraph 2, When the above instructions are executed individually or collectively by the at least one processor, the robot, A robot that inputs the identified plating information and the target feature information into a learned neural network model to identify feedback information corresponding to the plating.

4. In Paragraph 1, The above-mentioned at least one sensor is, Including a temperature sensor; further The above feature information is, It includes temperature information of the above-mentioned cooking material, When the above instructions are executed individually or collectively by the at least one processor, the robot, Identifying temperature information of the food based on sensing data obtained through the temperature sensor, and A robot that, when feedback information corresponding to the temperature is identified by comparing the identified temperature information and target feature information corresponding to the cooking material, transmits a control command corresponding to the feedback information to the external cooking device through the communication circuit.

5. In Paragraph 1, The above-mentioned at least one sensor is, It further includes an olfactory sensor, The above feature information is, It includes aroma information of the above-mentioned food, and When the above instructions are executed individually or collectively by the at least one processor, the robot, Identifying aroma information of the food based on sensing data obtained through the above olfactory sensor, and A robot that, when feedback information corresponding to the scent is identified by comparing the identified scent information and target feature information corresponding to the cooking material, transmits a control command corresponding to the feedback information to the external cooking device through the communication circuit.

6. In Paragraph 1, It further includes a driving unit; and When the above instructions are executed individually or collectively by the at least one processor, the robot, A robot that controls the drive unit to perform a cleaning operation on the hand of the robot.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the robot, A robot that updates target feature information of the food based on user feedback information regarding the food.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the robot, When the cooking state of the food is updated based on the above feedback information, the characteristic information of the updated food is identified, and A robot that provides cooking evaluation information obtained based on the above-mentioned updated cooking characteristic information.

9. In Paragraph 8, The above memory is, It further stores motion information corresponding to multiple emotional expressions, and When the above instructions are executed individually or collectively by the at least one processor, the robot, If any one of the plurality of emotional expressions is identified based on the above cooking evaluation information, operation information corresponding to the identified emotional expression is identified based on the information stored in the memory, and A robot that performs an action corresponding to the above-mentioned identified action information.

10. In Paragraph 1, The above robot is, A robot implemented as a dual-arm robot.

11. In the method of operation of the robot, An operation of identifying feedback information corresponding to the cooking state of a food item based on characteristic information of the food item identified based on sensing data acquired through at least one sensor and target characteristic information corresponding to the food item; and A method of operation comprising: transmitting a control command corresponding to the above-mentioned identified feedback information to an external cooking device.

12. In Paragraph 11, The above-mentioned at least one sensor is, Including a camera sensor; further The above feature information is, It includes plating information for the above-mentioned dish, The above method of operation is, An operation of identifying plating information of the food based on sensing data acquired through the camera sensor; and A method of operation comprising: comparing the identified plating information and target feature information corresponding to the cooked object, and when feedback information corresponding to the plating is identified, performing a feedback operation for the cooked object based on the feedback information.

13. In Paragraph 12, The operation of identifying the above feedback information is, A method of operation for identifying feedback information corresponding to the plating by inputting the identified plating information and the target feature information into a learned neural network model.

14. In Paragraph 11, The above-mentioned at least one sensor is, Including a temperature sensor; further The above feature information is, It includes temperature information of the above-mentioned cooking material, The above method of operation is, An operation of identifying temperature information of the food based on sensing data obtained through the temperature sensor; and A method of operation comprising: comparing the identified temperature information and target feature information corresponding to the cooking material, and when feedback information corresponding to the temperature is identified, transmitting a control command corresponding to the feedback information to the external cooking device.

15. One or more non-transient computer-readable storage media storing one or more computer programs comprising computer-executable instructions that cause said robot to perform operations when executed individually or collectively by at least one processor of said robot, wherein said operations are: An operation of identifying feedback information corresponding to the cooking state of a food item based on characteristic information of the food item identified based on sensing data acquired through at least one sensor and target characteristic information corresponding to the food item; and A storage medium comprising the operation of transmitting a control command corresponding to the above-mentioned identified feedback information to an external cooking device.

Images