Electronic apparatus, electronic apparatus operation method, and storage medium

Abstract

According to one aspect of the present disclosure, an electronic apparatus comprises: a display for displaying a three-dimensional space screen including a virtual space representing an indoor space and an object corresponding to a home appliance disposed in the indoor space; and a processor for displaying, on the display, a recipe card including recipe information. The processor controls so that the recipe information is transmitted to a prescribed home appliance when the recipe card is dragged to a prescribed area in the virtual space, and controls so that detailed information about the recipe information is displayed when the recipe card is tapped.

Description

Electronic device, method of operation of electronic device, and storage medium

[0001] The present disclosure relates to an electronic device and a method of operating the same, and more specifically to an electronic device capable of improving user convenience, a method of operating the electronic device, and a storage medium storing the method of operating the electronic device.

[0002] Home appliances can be electronic devices used inside homes or buildings, such as clothing processing units (washers, dryers, etc.), refrigerators, air conditioners, and cooking appliances. As the functionality of home appliances becomes more advanced, various features are being added to them. Furthermore, home appliances can provide various additional functions and services by being equipped with communication modules to communicate with other devices.

[0003] For example, prior art (Korean Patent Publication No. 10-2024-0120347) discloses a device and method for a user to control various smart devices placed in an indoor space through a user interface of a portable terminal. The prior art can display visual information of home appliances as a 3D animation and control the operation of home appliances on a 3D map.

[0004] The prior art can sequentially display at least one subtask of at least one home appliance to perform at least one of a cooking scenario, a washing scenario, and a cleaning scenario.

[0005] However, the prior art had the disadvantage that subtasks were displayed sequentially to execute the scenario, allowing only one home appliance to be controlled at a time.

[0006] In addition, prior art can display the indoor space state and the state of home appliances in various ways, such as text, numbers, colors, icons, and graphs on a 3D map, but does not display information about content related to home appliances.

[0007] Recently, an environment has been established where various cooking content and recipe information can be conveniently accessed through smartphone applications, enabling users to easily try new dishes. However, these mobile-based recipe services entail various inconveniences when frequently referring to or manipulating the smartphone screen during the actual cooking process.

[0008] In particular, recent cooking processes have evolved beyond simple cooking steps to involve the combined use of various dining appliances such as ovens, induction cooktops, microwaves, and air fryers, and there are many cases where the temperature, time, and mode of each appliance must be adjusted separately according to the recipe.

[0009] As a result, users may have difficulty manually setting and operating the appliance according to the cooking sequence, or problems may arise where the food is not cooked properly due to operational errors.

[0010]

[0011] The present disclosure aims to solve the aforementioned problems and other problems.

[0012] The purpose of the present disclosure is to provide an electronic device and a method of operating the same that can improve user convenience.

[0013] The purpose of the present disclosure is to provide a user experience that allows for convenient utilization of recipe information.

[0014] The objective of the present disclosure is to provide an electronic device and a method of operation thereof that can simultaneously improve cooking convenience and accuracy by linking recipe information to a dining-related home appliance and automatically transmitting it.

[0015] The objective of the present disclosure is to provide an electronic device and a method of operation thereof that allows a user to easily control the operation of a dining-related home appliance without having to manually operate it.

[0016] The objective of the present disclosure is to provide an electronic device and a method of operation thereof that can simultaneously apply settings optimized for a selected dish to each of a plurality of dining-related home appliances.

[0017] The objective of the present disclosure is to provide an electronic device and a method of operation thereof that can check content information about a recipe being cooked while checking the status of a home appliance.

[0018] The present disclosure provides a user experience in which a user can automatically transmit and control the corresponding recipe information to a plurality of associated dining-related home appliances or check detailed information through intuitive actions on a 3D space screen.

[0019] The problems of the present disclosure are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0020]

[0021] To achieve the above or other purposes, an electronic device and a method of operating the same according to one aspect of the present disclosure improve ease of use by providing a three-dimensional spatial screen that can check and control the status of a home appliance.

[0022] To achieve the above or other purposes, an electronic device and a method of operation according to one aspect of the present disclosure provide recipe information together with a three-dimensional spatial screen, so that recipe information can be easily utilized.

[0023] To achieve the above or other purposes, an electronic device according to one aspect of the present disclosure comprises: a display that displays a three-dimensional spatial screen including a virtual space representing an indoor space and an object corresponding to a home appliance placed in the indoor space; and a processor that displays a recipe card including recipe information on the display.

[0024] The processor controls the recipe information to be transmitted to a predetermined home appliance when there is an input in which the recipe card is dragged and dropped to a predetermined area in the virtual space.

[0025] The above processor controls the display of detailed information of the recipe information when there is an input of tapping the recipe card.

[0026] The processor can display the recipe card at the bottom of the virtual space when one or more dining-related home appliances exist in the virtual space.

[0027] The processor above provides N recipe cards that can be explored by swiping, and if the number of recipe cards set by the user is less than N, the remaining recipe cards can be provided as recommended recipe cards.

[0028] The recipe card set by the user above can be placed ahead of the recommended recipe card above.

[0029] The above processor can change the display state of an object corresponding to a home appliance capable of transmitting the recipe information while the recipe card is being dragged.

[0030] The processor may display additional dining icons when the entire virtual space or a portion of the virtual space is displayed.

[0031] The processor can control the switching to a dining screen in which all dining home appliances placed in the indoor space are displayed in a single virtual space when the dining icon is selected.

[0032] The above processor can control the dining home appliances placed in the indoor space to switch to a dining screen displayed in a single virtual space when the recipe information is transmitted.

[0033] A method of operating an electronic device according to one aspect of the present disclosure to achieve the above or other purposes comprises: displaying a three-dimensional space screen including a virtual space representing an indoor space and an object corresponding to a home appliance placed in the indoor space; displaying a recipe card including recipe information; transmitting the recipe information to a predetermined home appliance when there is an input of dragging the recipe card to a predetermined area in the virtual space; and displaying detailed information of the recipe information when there is an input of tapping the recipe card.

[0034] A storage medium according to one aspect of the present disclosure stores a computer-readable program for performing a method of operating an electronic device, and the method of operating the electronic device includes the steps of: displaying a three-dimensional space screen including a virtual space representing an indoor space and an object corresponding to a home appliance placed in the indoor space; displaying a recipe card including recipe information; transmitting the recipe information to a predetermined home appliance when there is an input of dragging the recipe card to a predetermined area in the virtual space; and displaying detailed information of the recipe information when there is an input of tapping the recipe card.

[0035]

[0036] According to at least one of the embodiments of the present disclosure, an electronic device and a method of operating the same can be provided to improve user convenience.

[0037] According to at least one of the embodiments of the present disclosure, a user experience that allows for convenient use of recipe information can be provided.

[0038] According to at least one of the embodiments of the present disclosure, cooking convenience and accuracy can be simultaneously improved by linking recipe information to a dining-related home appliance and automatically transmitting it.

[0039] According to at least one of the embodiments of the present disclosure, the operation of a dining-related home appliance can be easily controlled in conjunction without the user having to operate it individually.

[0040] According to at least one of the embodiments of the present disclosure, settings optimized for a selected dish can be applied at once to each of a plurality of dining-related home appliances.

[0041] According to at least one of the embodiments of the present disclosure, while checking the status of the home appliance, content information regarding the recipe being cooked can also be checked.

[0042] According to at least one of the embodiments of the present disclosure, a user experience can be provided in which the user can automatically transmit and control the corresponding recipe information to a plurality of associated dining-related home appliances or check detailed information through intuitive actions on a 3D space screen.

[0043] Meanwhile, various other effects will be disclosed directly or implicitly in the detailed description according to the embodiments of the present disclosure to be described below.

[0044] FIG. 1 is a configuration diagram of a home appliance system according to one embodiment of the present disclosure.

[0045] FIG. 2a is a block diagram of the main configuration of a server according to one embodiment of the present disclosure.

[0046] FIG. 2b shows an AI server according to one embodiment of the present disclosure.

[0047] FIG. 2c is an internal block diagram of a mobile terminal according to one embodiment of the present disclosure.

[0048] FIG. 3 is a drawing illustrating a method of operation of a home appliance system according to one embodiment of the present disclosure.

[0049] FIG. 4 is a flowchart illustrating a method of operation of an electronic device according to one embodiment of the present disclosure.

[0050] FIGS. 5 to 27 are drawings referenced in the description of the method of operation of an electronic device according to an embodiment of the present disclosure.

[0051] FIG. 28 is a flowchart illustrating a method of operation of an electronic device according to one embodiment of the present disclosure.

[0052] Embodiments of the present disclosure will be described in detail below with reference to the attached drawings. However, the present disclosure is not limited to these embodiments and can be modified in various forms.

[0053] In the drawings, parts unrelated to the description have been omitted to clearly and briefly explain the present disclosure, and the same drawing reference numerals are used for identical or extremely similar parts throughout the specification.

[0054] Meanwhile, the suffixes "module" and "part" for components used in the following description are assigned solely for the ease of drafting this specification and do not inherently confer any particularly significant meaning or role. Accordingly, the terms "module" and "part" may be used interchangeably.

[0055] Additionally, in this specification, terms such as first, second, etc. may be used to describe various elements, but these elements are not limited by these terms. These terms are used only to distinguish one element from another.

[0056] FIG. 1 is a configuration diagram of a home appliance system according to one embodiment of the present disclosure.

[0057] Referring to FIG. 1, a home appliance system according to one embodiment of the present disclosure includes a home appliance (20), a server (10), and a terminal (30).

[0058] Home appliances (20) are electronic devices for users, such as a washing machine (20a), a clothes processing device such as a dryer, an air conditioner such as an air purifier (20b), a refrigerator (20c), a water purifier (20e), a cooking appliance (20f), a vacuum cleaner (not shown), etc.

[0059] The home appliance (20) is equipped with a communication interface (not shown) internally and can exchange data with electronic devices within an internal network or with electronic devices accessible through an external network. To this end, the communication interface (not shown) can perform data exchange via wired or wireless means with a server (10), an AP device (40), etc.

[0060] The terminal (30) is a communication-capable electronic device, and may be a mobile terminal that a user can carry and move, such as a smartphone or a tablet PC. Additionally, the terminal (30) may be a terminal used in a fixed location, such as a desktop PC (not shown). To facilitate the provision of personalized information and services, it may be more preferable for the terminal (30) to be a mobile terminal (30) that an individual user carries and uses. The following describes embodiments in which a mobile terminal (30) is used as the terminal (30).

[0061] Meanwhile, the home appliance management application is an application for registering and controlling the operation of home appliances, and can be installed on a mobile terminal (30). The home appliance management application may be named ThinQ app.

[0062] The mobile terminal (30) can perform wireless communication with the home appliance (20), remotely control the wirelessly connected home appliance (20), or receive monitoring data from the home appliance (20).

[0063] The mobile terminal (30) can attempt to connect with the home appliance (20) using a SoftAP (Software enabled Access Point). A SoftAP may mean implementing the functions of a wireless access point in software so that it operates like a wireless AP.

[0064] The mobile terminal (30) can attempt to connect with the home appliance (20) by using a wireless LAN module to run the SoftAP.

[0065] The server (10) can transmit and receive data with a home appliance (20) connected via a network, etc. For example, the server (10) can provide update information to the home appliance (20) upon a request from a mobile terminal (30).

[0066] As another example, the server (10) can provide update information to the home appliance (20) when necessary, even if there is no request from the mobile terminal (30) or the home appliance (20).

[0067] Meanwhile, the home appliance (20) can also be upgraded with various settings (specific functions, detailed settings for each function, sound effects, operation sounds, etc.). Accordingly, the home appliance (20) can provide various effects to the user without limitation by receiving data for new settings from the server (10) or the mobile terminal (30) and upgrading them. Therefore, in addition to the settings provided at the time of release of the home appliance (20) product, various functions and effects can be provided.

[0068] The home appliance (20) can run system applications, applications related to operating modes, or download applications internally.

[0069] Meanwhile, the update information may include firmware update information, software update information, etc. The firmware update information is data for updating the firmware within the home appliance (20), and may, for example, be update information regarding load control or safety control of the home appliance (20). Meanwhile, the software update information may include upgrade information related to a new operating mode. Based on this upgrade information related to the new operating mode, an upgrade of the operating mode may be performed. Meanwhile, the software update information may further include update information regarding system applications, applications related to the operating mode, or download applications. Based on this update information, an update of the operating mode may be performed. The firmware update information may include data data regarding the firmware, operating system, or framework. Based on this firmware update information, the firmware may be updated.

[0070] Meanwhile, the home appliance system may further include an access point (AP) device (40). The AP device (40) can provide an internal network to adjacent electric devices. In particular, it can provide a wireless network.

[0071] Meanwhile, the AP device (40) can allocate a wireless channel according to a predetermined communication method to electronic devices within an internal network and perform wireless data communication through the said channel. Here, the predetermined communication method may be a WiFi communication method.

[0072] At this time, a mobile terminal (30) located within the internal network can perform monitoring, remote control, etc. of the home appliance (20) by connecting to the home appliance (20) through the AP device (40).

[0073] Meanwhile, the AP device (40) can perform data communication with external electronic devices through an external network in addition to the internal network. For example, the AP device (40) can perform wireless data communication with a mobile terminal (30) located outside through an external network. At this time, the mobile terminal (30) located in the external network can perform monitoring, remote control, etc. of the home appliance by connecting to the home appliance (20) through the external network and the AP device (40).

[0074] As another example, the AP device (40) can perform wireless data communication with a server (10) located outside through an external network.

[0075] Meanwhile, the server (10) can register product information for the home appliance (20). The server (10) can store firmware information and operation information (course information, etc.) for the home appliance (20). For example, the server (10) may be a server operated by the manufacturer or seller of the home appliance (20).

[0076] FIG. 2a is a block diagram of the main configuration of a server according to one embodiment of the present disclosure.

[0077] Referring to FIG. 2a, the server (10) includes a communication interface (120), memory (130), and a processor (110).

[0078] The communication interface (120) can communicate with a mobile terminal (30), a home appliance (20), or another server. For example, communication technologies used by the communication interface (120) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), etc.

[0079] The memory (130) can store various data, such as data received from external devices like a home appliance (20), a terminal (30), and another server, and data generated within the server (10).

[0080] Additionally, the memory (130) can store information related to the home appliance (20). Additionally, the memory (130) may store MAC addresses of multiple home appliance (20) products and multiple serial numbers corresponding to each of the multiple MAC addresses.

[0081] The processor (110) can control the overall operation of the server.

[0082] According to an embodiment, the processor (110) can generate an automatic registration key for automatic registration of the home appliance (20).

[0083] The processor (110) can determine whether the home appliance (20) is registered.

[0084] A home appliance (20) that wishes to receive remote services from a server (10) may be registered with and managed by the server (10) with its own unique device ID. The server (10) can identify the home appliance (20) through the device ID. The server (10) may have product information (e.g., product type, model information, etc.) of the home appliance (20) registered together with the device ID. Here, the device ID may be created independently of product characteristic information such as product type or model information, but the product characteristic information may be coded together. Thus, the server (10) may be designed to recognize the product type or model information using only the device ID.

[0085] The user can request various services from the server (10) through a mobile terminal (30) that the user always carries, such as a smartphone. To do this, a home appliance management application capable of implementing this can be installed on the mobile terminal (30). The home appliance management application (LG ThinQ app) can provide a service that allows controlling home appliances (20) and various parts of the house anytime and anywhere with a single application.

[0086] The mobile terminal (30) is equipped with various input / output means, so the user can very conveniently request various services.

[0087] Product registration of the home appliance (20) is the starting point of the home appliance management service, and the user can also perform product registration of the home appliance (20) using a mobile terminal (30).

[0088] FIG. 3 is a diagram illustrating the operation method of a home appliance system according to one embodiment of the present disclosure, and is a signal flow diagram of a home appliance (20), a mobile terminal (30), an access point (AP) device (40), and a server (10).

[0089] Referring to FIG. 3, the mobile terminal (30) attempts to connect to the home appliance (20) in access point (AP) mode (S210), and a communication connection can be established between the mobile terminal and the communication module (e.g., Wi-Fi modem) of the home appliance (20) (S215).

[0090] In addition, the home appliance (20) can be switched to AP mode by operating a specific button of the home appliance (20) in advance.

[0091] Meanwhile, the mobile terminal (30) can receive information (identification information (SSID), password information, etc.) of the AP device (40) to be registered (S220).

[0092] When the user manually enters the information of the AP device (40) to be registered (S220), the mobile terminal (30) can transmit the information of the AP device (40) to be connected to the home appliance (20) (S225).

[0093] Additionally, the mobile terminal (30) can request the server (10) to register the product of the home appliance (20) (S230).

[0094] According to an embodiment, the mobile terminal (30) can request product registration of the home appliance (20) by transmitting product information of the home appliance (20) and / or information of the AP device (40) to be registered to the server (10) (S230).

[0095] Meanwhile, the home appliance (20) requests a connection to the access point device (40) using the information of the received AP device (40) (S240), and the access point device (40) may approve the connection based on the received identification information and password information (S245).

[0096] And, after connecting with the access point device (40), the home appliance (20) can request product registration and server connection to the server (10) via the access point device (40) (S250).

[0097] At this time, the home appliance (20) can transmit product information to the server (10). In addition, according to an embodiment, the home appliance (20) can also transmit information of the access point device (40) and / or information of the mobile terminal (30) to the server (10).

[0098] When the server (10) receives a registration request successfully from both the home appliance (20) and the mobile terminal (30), it can complete the product registration using the received product information and send a product completion message to the mobile terminal (30) via the access point device (40) (S255). Accordingly, the product registration of the home appliance can be easily performed.

[0099] Meanwhile, the server (10) and / or mobile terminal (30) may be an AI device including a machine-learned artificial neural network. For example, the server (10) and / or mobile terminal (30) may include an artificial neural network that identifies a product family of home appliances (20) based on images. Additionally, the server (10) and / or mobile terminal (30) may include an artificial neural network that identifies a home appliance associated with recipe information.

[0100] Artificial Intelligence (AI) refers to the field of researching artificial intelligence or the methodologies to create it, while Machine Learning refers to the field of researching methodologies to define and solve various problems addressed within the field of AI. Machine Learning is also defined as an algorithm that improves performance on a task through continuous experience.

[0101] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to any model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses. An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0102] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.

[0103] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters, on the other hand, refer to parameters that must be set prior to training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0104] The objective of training an artificial neural network can be viewed as determining model parameters that minimize the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process of an artificial neural network.

[0105] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0106] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result) that the neural network must infer when training data is input. Unsupervised learning refers to a method of training an artificial neural network without labels provided for the training data. Reinforcement learning refers to a learning method in which an agent defined within an environment is trained to select an action or sequence of actions that maximizes the cumulative reward in each state.

[0107] Machine learning implemented using a Deep Neural Network (DNN) that includes multiple hidden layers among artificial neural networks is also called Deep Learning, and Deep Learning is a part of Machine Learning. Hereinafter, Machine Learning is used in a sense that includes Deep Learning.

[0108] FIG. 2b shows an AI server according to one embodiment of the present disclosure.

[0109] Referring to FIG. 2b, the AI ​​server may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network. Here, the AI ​​server may be composed of multiple servers to perform distributed processing, or it may be defined as a communication network. In this case, the AI ​​server may be included as part of the configuration of a server (10) or a mobile terminal (20) to perform at least part of the AI ​​processing together.

[0110] The AI ​​server may include a communication interface (120), memory (130), a learning processor (140), and a processor (110), etc.

[0111] The communication interface (120) can transmit and receive data with other devices, such as a mobile terminal (30).

[0112] The memory (130) may include a model storage unit (131). The model storage unit (131) may store a model (or artificial neural network, 231a) that is being learned or has been learned through a learning processor (140).

[0113] The learning processor (140) can train a model composed of an artificial neural network using training data. Here, the trained artificial neural network may be referred to as a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for judgment to perform an action.

[0114] The learning processor (140) can train the artificial neural network (131a) using training data. The training model may be used while mounted on the AI ​​server of the artificial neural network, or it may be used while mounted on another device.

[0115] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (130).

[0116] The processor (110) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0117] The server (10) can receive recipe information, information about the home appliance (20), etc. through the communication interface (120).

[0118] The above server (10) may be equipped with an artificial neural network (131a) that independently determines a home appliance related to each recipe information. Additionally, the artificial neural network (131a) may determine the optimal setting value of the home appliance for each recipe information.

[0119] FIG. 2c is an internal block diagram of a mobile terminal according to one embodiment of the present disclosure.

[0120] The communication unit (210) can transmit and receive data with external devices, such as other AI devices or AI servers, using wired or wireless communication technology. For example, the communication unit (210) can transmit and receive sensor information, user input, learning models, control signals, etc., with external devices.

[0121] At this time, the communication technologies used by the communication unit (210) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0122] Referring to FIG. 2c, the input unit (220) may include a camera (Camera, 221) for inputting a video signal, a microphone (Microphone, 222) for receiving an audio signal, and a user input unit (User Input Unit, 223) for receiving information from a user.

[0123] Voice data or image data collected from the input unit (220) can be analyzed and processed into user control commands.

[0124] The input unit (220) is for inputting video information (or signal), audio information (or signal), data, or information input from a user, and for inputting video information, the input unit (220) may be equipped with one or more cameras (221).

[0125] The camera (221) processes image frames, such as still images or video, obtained by an image sensor in video call mode or shooting mode. The processed image frames may be displayed on a display unit (Display Unit, 251) or stored in memory (270).

[0126] The microphone (222) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application running) being performed on the AI ​​device. Meanwhile, various noise removal algorithms can be applied to the microphone (222) to remove noise generated during the process of receiving external acoustic signals.

[0127] The user input unit (223) is for receiving information from a user, and when information is input through the user input unit (223), the processor (280) can control the operation of the mobile terminal (30) to correspond to the input information.

[0128] The user input unit (223) may include mechanical input means (or mechanical keys, such as buttons, dome switches, jog wheels, jog switches, etc. located on the front / rear or side of the mobile terminal (30)) and touch input means. As an example, the touch input means may consist of a virtual key, soft key, or visual key displayed on a touchscreen through software processing, or a touch key placed on a part other than the touchscreen.

[0129] The output unit (250) may include at least one of a display unit (Display Unit, 251), a sound output unit (Sound Output Unit, 252), a haptic module (Haptic Module, 253), and an optical output unit (Optical Output Unit, 254).

[0130] The display unit (251) displays (outputs) information processed by the mobile terminal (30). For example, the display unit (251) can display information on the execution screen of an application running on the mobile terminal (30), or UI (User Interface) and GUI (Graphic User Interface) information based on such execution screen information.

[0131] The display unit (251) can implement a touch screen by forming a layered structure with the touch sensor or by being formed as an integral unit. This touch screen functions as a user input unit (223) that provides an input interface between the mobile terminal (30) and the user, and at the same time, can provide an output interface between the terminal (30) and the user.

[0132] The sound output unit (252) can output audio data received from the communication unit (210) or stored in the memory (270) in call signal reception, call mode or recording mode, voice recognition mode, broadcast reception mode, etc.

[0133] The sound output unit (252) may include at least one of a receiver, a speaker, and a buzzer.

[0134] The haptic module (253) generates various tactile effects that the user can feel. A typical example of the tactile effect generated by the haptic module (253) can be vibration.

[0135] The light output unit (254) outputs a signal to indicate the occurrence of an event using the light from the light source of the AI ​​device. Examples of events occurring in the mobile terminal (30) may include receiving a message, receiving a call signal, a missed call, an alarm, a schedule notification, receiving an email, receiving information through an application, etc.

[0136] According to an embodiment, the mobile terminal (30) may also include a learning processor (230), and the learning processor (230) may train an artificial neural network (not shown) stored in memory (270).

[0137] The sensing unit (240) can acquire at least one of internal information of the mobile terminal (30), surrounding environment information of the mobile terminal (30), and user information using various sensors. At this time, the sensors included in the sensing unit (240) include a proximity sensor, an illuminance sensor, an accelerometer, a magnetic sensor, a gyroscope, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor, a microphone, a lidar, a radar, etc.

[0138] In an embodiment where the mobile terminal (30) directly identifies the home appliance (20), the processor (280) can identify the home appliance associated with the recipe information. An artificial neural network (131a) may be provided in the learning processor (230) or memory (270). The processor (280) inputs the recipe information into the artificial neural network (131a) and can identify the home appliance associated with the recipe information based on the output of the artificial neural network (131a).

[0139] FIG. 4 is a flowchart illustrating a method of operation of an electronic device according to one embodiment of the present disclosure. FIG. 5 to 27 are drawings referenced in the description of a method of operation of an electronic device according to an embodiment of the present disclosure.

[0140] Referring to FIG. 4, the electronic device can display a three-dimensional (3D) spatial screen (S410). The electronic device may be a user terminal. For example, a mobile terminal (30) may be an electronic device that provides a three-dimensional spatial screen that can check and control the status of a home appliance (20).

[0141] The three-dimensional space screen includes a virtual space representing an indoor space and an object corresponding to a home appliance (20) placed in the indoor space.

[0142] The processor (280) can display a three-dimensional spatial screen including virtual space and home appliance objects on the display (251).

[0143] A home appliance object may correspond to a home appliance (20). For example, the home appliance object may be a graphic object such as an icon or image representing the corresponding home appliance (20).

[0144] The virtual space may be based on an indoor space where the home appliance (20) is placed. For example, the virtual space may correspond to the user's entire house or a designated room, and the 3D space screen may be a 3D home view screen that represents the user's entire house or a designated room in three dimensions.

[0145] A 3D space screen is a visual interface that represents space in three dimensions and may also include a top view from a perspective looking from above.

[0146] The three-dimensional space screen may be divided into multiple areas, such as multiple rooms, a kitchen, and a living room. The three-dimensional space screen may include a full screen displaying the entire area and an area screen displaying one or more selected areas. The processor (280) may switch between displaying the full screen and the area screen according to settings or user input.

[0147] The full screen may be a top view that makes it easy to grasp the overall structure of the indoor space and the placement status of home appliances. The area screen may be an immersive view that displays a specific area as a three-dimensional image.

[0148] Additionally, the processor (280) can receive information about the indoor space and information about the home appliance (30) through user input or the server (10). Additionally, the processor (280) can receive mapping information of the home appliance (30) placed in the indoor space.

[0149] The processor (280) can place an object corresponding to the home appliance (20) in a virtual space by reflecting the location of the home appliance (20) placed in an indoor space based on the received information. The processor (280) can visualize and display the home appliance (20) on a three-dimensional space screen.

[0150] The mobile terminal (30) can receive status information of the home appliance (20) through the communication unit (210). The mobile terminal (30) can directly receive status information from the home appliance (20) through the communication unit (210). Alternatively, the home appliance (20) can transmit status information to the server (10), and the mobile terminal (30) can receive status information from the server (10) through the communication unit (210).

[0151] The processor (280) can control the display of at least one of the received status information. The status information of the home appliance (20) may be displayed superimposed on an object corresponding to each home appliance (20) or displayed at a predetermined location adjacent to the object. For example, the status information of the home appliance (20) may be displayed in the form of a bubble pop-up spaced apart from an object corresponding to the home appliance (20) in 3D space.

[0152] By displaying the status of each home appliance (20) on a three-dimensional space screen, the user can intuitively check the status of the home appliance (20) along with its position.

[0153] In addition, to increase the utility of the 3D space screen, more content related to the home appliance (20) can be provided. The 3D space screen is utilized as a space to display the home appliance (20) and content related to the home appliance (20). That is, the status of the home appliance (20) can be checked on a single screen, and content that can be transmitted to the home appliance (20) is displayed.

[0154] Content related to the home appliance (20) may be recipe information. For example, the content related to the home appliance (20) may be composed of recipes that can be sent to the home appliance (20), so that the user can display recipes that can be sent immediately.

[0155] For example, the mobile terminal (30) can provide recipe information along with a three-dimensional spatial screen (S420). The user can check the recipe information using the mobile terminal (30) and easily use the related home appliance.

[0156] The processor (280) can display a recipe card containing recipe information on the display (251) (S420). For example, the processor (280) can display a recipe card containing recipe information on the lower side of the three-dimensional space screen. In this case, the lower area where the recipe card is displayed may be named the recipe area, and the upper area where an object corresponding to the home appliance (20) is displayed may be named the product area.

[0157] The user can run a home appliance management application through a mobile terminal (30) and use a three-dimensional space screen and recipe information.

[0158] Additionally, the processor (280) can display a plurality of recipe cards, each containing recipe information, on the display (251) (S420).

[0159] Referring to FIG. 5, the user interface screen (500) of the home appliance management application displayed on the display (251) may include a three-dimensional space screen (510) and a plurality of recipe cards (520).

[0160] The three-dimensional space screen (510) exemplified in FIG. 5 may correspond to a kitchen space within the entire indoor space. Objects (511, 512, 513) corresponding to home appliances placed in the kitchen space may be displayed on the three-dimensional space screen (510).

[0161] For example, when a refrigerator, water purifier, and oven are placed and in use in an actual kitchen, the refrigerator object (511), water purifier object (512), and oven object (513) may be displayed on the 3D space screen (510).

[0162] Additionally, on the upper side of the refrigerator object (511), water purifier object (512), and oven object (513), the status information of the refrigerator (511a), the status information of the water purifier (512a), and the status information of the oven (513a) may be displayed, respectively.

[0163] In another embodiment, the three-dimensional space screen (510) illustrated in FIG. 5 may be configured as a virtual kitchen space instead of representing the user's actual physical space. In this case, even if the user places dining-related home appliances in multiple rooms, all dining-related home appliances are visible in a single room on the user interface screen (500). By utilizing a virtual space, dining-related home appliances placed in multiple rooms can be monitored at a glance and the space can be represented in a new way.

[0164] Meanwhile, in this specification, 'dining' may refer to a space for eating or the meal itself. 'Dining-related home appliances' may refer to home appliances used in a space (such as a dining room or a combined kitchen and dining area) related to meal preparation, eating, and cleanup.

[0165] For example, 'dining-related home appliances' or 'dining home appliances' may include home appliances related to meals and meal preparation, such as microwave ovens, induction cooktops, ovens, water purifiers, and beverage dispensers, as well as home appliances related to organization, such as dishwashers.

[0166] In this specification, 'dining-related home appliance' or 'dining home appliance' may refer to a home appliance product capable of transmitting recipes. For example, a microwave oven (a model capable of transmitting recipes), an electric range (a model capable of transmitting recipes), a water purifier (a model capable of transmitting recipes), etc., may be considered a dining-related home appliance.

[0167] Meanwhile, the processor (280) can display a recipe card (520) containing recipe information (S420).

[0168] The processor (280) can provide a dining service including recipe cards in one area of ​​the user interface screen (500). The dining service may consist of M recipe cards, excluding recipes created by me (the user). The dining service provides N (e.g., 10) recipe cards at a time and can be navigated by swiping left and right.

[0169] The recipe card included in the dining service can be selected from recipes that can be transmitted to a home appliance placed in the indoor space.

[0170] Meanwhile, the user may not register all home appliances placed in the indoor space in the virtual space of the appliance management application. Therefore, the recipe cards included in the dining service can be selected from among recipes that can be transmitted to home appliances placed in the virtual space.

[0171] The default display order of multiple recipe cards may be the order of the favorite recipe card (521) registered by the user and the recommended recipe card (522).

[0172] If there are fewer than N favorite recipes (e.g., 10), the remaining recipe cards (522) are provided as recommended recipes.

[0173] In FIG. 5, two favorite recipe cards (521a, 521b) are registered and displayed first. After the two favorite recipe cards (521a, 521b), N-2 recommended recipe cards (522a, 522b, 522c, 522d, ...) are placed.

[0174] Meanwhile, the recommended recipe cards (522a, 522b, 522c, 522d, ...) can provide recipes that can be sent to home appliances placed by the user in a virtual space in a random order. Additionally, the recommended recipe cards (522a, 522b, 522c, 522d, ...) can be refreshed whenever the home appliance management application is launched.

[0175] Alternatively, even if the user does not register a favorite, recommended recipe cards (522a, 522b, 522c, 522d, ...) can be determined based on the order of usage frequency.

[0176] Users can conveniently browse favorited recipes, frequently used recipes, and recommended recipes.

[0177] In addition, while checking the status of the home appliance, you can also view content information about the recipe being cooked.

[0178] The processor (280) and / or the learning processor (230) may include an artificial neural network that determines the home appliance associated with the recipe card (520).

[0179] Additionally, the processor (280) and / or the learning processor (230) may include an artificial neural network that determines a recipe that can be transmitted to a home appliance placed in a virtual space.

[0180] Meanwhile, dining services may be provided when at least one of the dining home appliance products is placed on the 3D space screen. Even if the dining home appliance products are registered in the appliance management application, dining services may not be provided if they are not placed on the 3D space screen.

[0181] In addition, if one dining home appliance product is placed in the virtual space of a 3D space screen, dining services can be provided in that area (or room).

[0182] When dining-related home appliance products are placed in multiple areas (or rooms) in a virtual space of a 3D space screen, dining services can be provided in the entire area (or room) where the dining home appliance products are placed.

[0183] Meanwhile, the user can command the transmission of a recipe on a three-dimensional space screen. If there is an input (S430, S440) for a recipe card, the processor (280) can control the communication unit (210) so that recipe information corresponding to the input recipe card is transmitted to a home appliance (20) associated with the recipe information (S450).

[0184] The mobile terminal (30) can directly transmit recipe information to the home appliance (20) through the communication unit (210).

[0185] Alternatively, the mobile terminal (30) can transmit recipe information to the server (10) through the communication unit (210), and the home appliance (20) can receive recipe information from the server (10).

[0186] Alternatively, the mobile terminal (30) can transmit information to the server (10) regarding what recipe information the user has commanded to be transmitted and what the home appliance product to be transmitted is, and the server (10) can transmit the corresponding recipe information to the corresponding home appliance (20) based on the received information.

[0187] The processor (280) can control that when any one of the multiple recipe cards is selected, recipe information corresponding to the selected recipe card is transmitted to multiple home appliances associated with the selected recipe card. That is, the recipe information to be input can be transmitted to multiple products at once.

[0188] When a drag and drop input is received in which one of the plurality of recipe cards is dragged and the drag is terminated in a predetermined area, the processor (280) can control the transmission of recipe information corresponding to the selected recipe card to a plurality of home appliances associated with the selected recipe card.

[0189] A plurality of home appliances associated with the above-mentioned selected recipe card may include one or more home appliances placed in the above-mentioned predetermined area. That is, recipe information can be transmitted to the home appliance placed in the area where the drag was basically terminated, and recipe information can also be transmitted to home appliances placed in other areas according to the recipe information.

[0190] Meanwhile, the processor (280) can control the recipe information to be transmitted immediately when there is input for the recipe card.

[0191] According to an embodiment, the processor (280) can control the transmission of recipe information with a time difference when there are time-consuming steps in the cooking process, such as preparing ingredients. For example, the processor (280) can control the transmission of recipe information to two home appliances with a time difference. Alternatively, the processor (280) can control the transmission of recipe information to a home appliance in part, and then transmit the recipe information to the same home appliance after a certain period of time.

[0192] The home appliance associated with the above recipe information may be a dining home appliance associated with at least one of the cooking processes included in the above recipe information.

[0193] The above recipe information can be configured to automatically set the control parameters of a home appliance associated with the recipe information to match the recipe information. A dining home appliance that receives the recipe information is configured according to the recipe. When the recipe is transmitted, the home appliance is automatically configured, and cooking is possible with optimized modes, temperatures, cooking times, and water output without product operation.

[0194] Therefore, settings optimized for the selected dish can be applied to each of the multiple inning home appliances at once.

[0195] Users can identify the device to which the recipe is being sent through visualization elements displayed in a 3D space, set the device to cooking values ​​optimized for each device, and check the cooking status through device monitoring. Therefore, steps from the pre-cooking stage to the post-cooking stage can be handled on a single screen.

[0196] By providing a recipe transfer function on the screen where home appliance objects are displayed, convenience can be enhanced and the utilization of recipes and home appliance products can be increased.

[0197] In addition, users can send recipes to multiple home appliances at once and control multiple home appliances at once with a single input to the recipe card.

[0198] Meanwhile, the processor (280) can update the status information of the home appliance after the transmission of the recipe information (S460). After the transmission of the recipe information, the processor (280) can change the status information of the home appliance that received the recipe information.

[0199] For example, if the oven receives recipe information, the oven's status information (513a) may be changed to 'transmission complete'. Additionally, depending on the cooking stage in progress, the oven's status information (513a) may be changed to 'cooking in progress', 'cooking completed', etc.

[0200] Additionally, the processor (280) can also update the recipe card after the recipe information is transmitted. For example, after the recipe information is transmitted, the processor (280) can display the cooking steps on the recipe card as one of transmission complete / cooking in progress / cooking complete.

[0201] For example, the oven can transmit mode information such as food drying, air frying, fermentation, steaming, grilling, and automatic cooking, as well as temperature, cooking time, and option information for each mode (e.g., cooking options such as steak or steamed eggs in automatic cooking). Additionally, the oven object's bubble popup can display status information such as waiting or offline (before sending a recipe), setting up (after sending a recipe), paused, cooking, or cooking completed (in use).

[0202] For example, the water purifier can transmit information such as capacity and water temperature. Additionally, the water purifier's bubble pop-up can display status information such as On or Off (before recipe transmission) or On (after recipe transmission).

[0203] For example, the electric range can transmit guidance information regarding burner mode and cooking time steps. Additionally, the bubble pop-up of the electric range can display status information such as on, off, or offline (before sending the recipe), setting (after sending the recipe), cooking, or cooking finished (while in use).

[0204] For example, washing course information can be transmitted to the dishwasher. Additionally, the dishwasher bubble popup can display status information such as on, off or offline (before transmitting the recipe), on (after transmitting the recipe), paused, washing, washing completed, canceling, and remaining time information during washing / rinsing / drying (in use).

[0205] First, a first input for the recipe card can be received (S430). Then, a second input for a specific area of ​​the indoor space can be received (S440).

[0206] The processor (280) can control the recipe information to be transmitted to a home appliance associated with the recipe information if there is a first input for the recipe card (S430) and a second input for a predetermined area of ​​the virtual space (S440) (S450).

[0207] The first input above may be an input for dragging the recipe card, and the second input may be an input for ending the drag in the predetermined area. That is, the first input and the second input may be drag and drop inputs for dragging the recipe card to a predetermined area in the virtual space and dropping it.

[0208] The processor (280) can change the display state of an object corresponding to a home appliance capable of transmitting the recipe information when there is a first input. That is, in the section where the drag starts and proceeds, the transmittable home appliance can be intuitively distinguished from the non-transmittable home appliance.

[0209] Users can check the home appliances linked to the recipe during the recipe transmission process. Users can also preview the home appliances that will be set up when the recipe is transmitted.

[0210] The processor (280) can control the transmission of a recipe card to an associated home appliance when there is an input of dragging and dropping the recipe card to a designated area in the virtual space. By automatically transmitting the recipe information to the associated home appliance, cooking convenience and accuracy can be improved simultaneously.

[0211] Additionally, the processor (280) can highlight the recipe card and the object corresponding to the home appliance that received the recipe information after the transmission of the recipe information.

[0212] FIG. 6 illustrates an example of a user interface screen before recipe information transmission, and FIG. 7 illustrates an example of a user interface screen after recipe information transmission.

[0213] Referring to FIGS. 6 and 7, home appliance objects (611, 612, 613) and status information (611a, 612a, 613a) of each home appliance are displayed on the three-dimensional space screen (610). Additionally, a plurality of recipe cards (621, 622, 623) may be displayed on the lower side of the three-dimensional space screen (610).

[0214] Home appliance objects (611, 612, 613) represent home appliances (refrigerators, water purifiers, cooking appliances) placed in an indoor space. Some home appliances may be composite devices that perform two or more functions. For example, a cooking appliance may include one or more ovens (or electric ranges) and a cooktop installed above them.

[0215] The user can create a representation similar to the actual appearance through the arrangement of home appliance objects (611, 612, 613) and furniture.

[0216] The status information (611a, 612a, 613a) of the home appliance represents the current state of the home appliance (20). For a dining-related home appliance, the status may be displayed as standby (power saving mode), on, product setup in progress, remaining time, transmission status, and cooking complete status.

[0217] The user can navigate recipe cards (621, 622, 623) by scrolling left and right. Recipe cards (621, 622, 623) may consist of a representative recipe photo, a recipe name, and a quantity adjustment button. According to an embodiment, one card (623) may be a card object that displays another recipe when selected.

[0218] Recipes set as favorites on recipe cards (621, 622, 623) are provided with a star mark and are provided in order before recipes that are not set as favorites.

[0219] When a user drags a desired recipe from the recipe cards (621, 622, 623) to the area where the home appliance objects (611, 612, 613) are displayed, the recipe can be transmitted to the associated home appliance.

[0220] According to an embodiment, when transmitting a recipe, the home appliance to which the recipe can be transmitted may be highlighted, etc. The home appliance to which the recipe can be transmitted varies depending on the recipe. Therefore, when a user transmits a recipe, it is possible to visually distinguish and indicate which home appliance can be transmitted to.

[0221] Referring to FIG. 7, after the recipe is transmitted, the recipe being cooked and the status information of the home appliance are displayed on a 3D space screen. For example, the status of the water purifier changes from the on state (612a) to the transmission complete state (612b). The status of the cooking appliance changes from standby (613a) to cooking remaining time information (613b).

[0222] The remaining time information (613b) can be updated as cooking progresses. Additionally, the water purifier's status can be changed back to the on state (612a).

[0223] Meanwhile, after the recipe is transmitted, the recipe cards (711, 712, 713) may also be updated. For example, the display order and display status of the transmitted recipe card (711) may be changed. Additionally, an indication that cooking is in progress may be provided on the transmitted recipe card (711).

[0224] Meanwhile, the processor (280) can control the transmission of the recipe information when the home appliance associated with the recipe information is turned on. However, the processor (280) can control the display of guidance information indicating the off state when the home appliance associated with the recipe information is turned off.

[0225] FIGS. 8 to 13 sequentially illustrate an example of a process for transmitting recipe information.

[0226] Referring to FIG. 8, the user interface screen (800) of the home appliance management application displayed on the display (251) may include a three-dimensional space screen (820).

[0227] When the home appliance management application is launched, the 'All' item (810) is selected as the default setting, and a three-dimensional space screen (820) representing the entire virtual space can be displayed.

[0228] For example, the three-dimensional space shown in Fig. 8 can utilize a top view that represents the entire space.

[0229] The three-dimensional space screen (820) may include a plurality of regions (821, 822, 823, 824). Home appliance objects may be placed in the plurality of regions (821, 822, 823, 824).

[0230] The user can select an area (or room) where the product to which recipe information is to be transmitted is placed. For example, the user can select the kitchen area (821) by touching it.

[0231] Referring to FIG. 9, you can enter an immersive view (900) of the selected area.

[0232] The immersive view (900) may include a three-dimensional spatial screen (910) representing a selected kitchen area within the entire indoor space and a recipe card (920).

[0233] Additionally, on the 3D space screen (910), objects (911, 912, 913) corresponding to home appliances placed in the kitchen space and status information (911a, 912a, 913a) of the home appliances can be displayed.

[0234] For example, when a refrigerator, water purifier, and oven are registered in the kitchen space, the refrigerator object (911) and the refrigerator's status information (911a), the water purifier object (912) and the water purifier's status information (912a), and the oven object (913) and the oven's status information (913a) may be displayed on the 3D space screen (910).

[0235] Multiple recipe cards (920) (921, 922, 923, 924) are displayed. Depending on the input of dragging and dropping any one of the multiple recipe cards (921, 922, 923, 924), the display state of the home appliance object (912, 913) associated with the recipe information of the corresponding recipe card (922) may be changed.

[0236] The changed display state of the home appliance object (912, 913) associated with the recipe information can be maintained while dragging the recipe card (922).

[0237] Referring to FIG. 10, while dragging the recipe card (922), the home appliance object (912, 913) associated with the recipe information may blink or be highlighted. However, if the home appliance product corresponding to the home appliance object (912, 913) is turned off or offline, it may not be highlighted or blinked.

[0238] Additionally, the highlight / blink display can be disabled when the screen is magnified. Recipe information can still be transmitted even when the screen is magnified.

[0239] When the drag is finished, the recipe information of the recipe card (922) can be transmitted to a home appliance associated with the recipe information.

[0240] When dragging and dropping a home appliance object (912, 913) into a product area, the recipe cooking value can be transferred to a product that is in a standby or turned-on state among the associated home appliances.

[0241] Referring to Fig. 11, at the moment the recipe card (922) is dropped, the recipe card (922) disappears and the recipe card (922) reappears in its original location.

[0242] Meanwhile, if there are no transferable products (all offline / off / cooking), a popup can indicate the product's off / offline status upon dropping.

[0243] Meanwhile, if a product that is in standby or turned on is placed in another room, it can be transferred through an additional product selection step.

[0244] For example, when an oven is placed in space A and another oven is placed in space B, a selection popup can be provided for which oven to send the recipe to when sending a lightwave oven recipe from A.

[0245] For example, if a water purifier is placed in space A and an oven is placed in space B, when a lightwave oven recipe is sent from A, a selection popup can be provided asking whether to send it to the lightwave oven in space B.

[0246] Figure 11 illustrates the state after recipe transmission is completed.

[0247] When recipe information is transmitted to the oven and settings based on the recipe information are being applied, the oven's status information can be changed from a standby state (913a) to a setting state (913b).

[0248] Meanwhile, in the case of a water purifier, the status information can be temporarily changed to transmission complete and then restored to the on state (912a).

[0249] Additionally, the display state of the recipe card (922) corresponding to the transmitted recipe information may also be changed. For example, the recipe card (922) may be highlighted and highlighted (922a). Additionally, status information such as "in progress" may be displayed on the recipe card (922).

[0250] FIG. 12 illustrates the state during cooking. Referring to FIG. 12, the state information of the oven during cooking displays remaining time information (913c).

[0251] FIG. 13 illustrates the cooking completion status. Referring to FIG. 13, the status information of the oven that has completed cooking changes to cooking complete (913d). Additionally, cooking completion status information (922b) may be displayed on the recipe card (922). The cooking completion status information (913d, 922b) may be provided only on the day and may not be provided on the next day.

[0252] Meanwhile, the mobile terminal (30) can provide a push notification when cooking is complete. The mobile terminal (30) can provide a push notification when the operation of a dining home appliance, such as a microwave oven or electric range, ends (e.g., 'Cooking is complete').

[0253] Meanwhile, the processor (280) controls the display of detailed information of the recipe information when there is an input to tap the recipe card.

[0254] Referring to FIG. 14 (a), home appliance objects and status information of each home appliance are displayed on the three-dimensional space screen (1410). Additionally, a plurality of recipe cards (1420) may be displayed on the lower side of the three-dimensional space screen (1410). According to an embodiment, one card (1423) may be a card object that displays another recipe when selected.

[0255] Users can tap or click the recipe card (1421, 1422) to view the detailed recipe.

[0256] Figure 14 (b) illustrates a detailed information screen (1430) of recipe information. The detailed information screen (1430) provides detailed information regarding the recipe cooking method and ingredients. When the back button is selected, you can move to the previous screen.

[0257] According to the embodiment, a button to send to one device, such as 'Send to oven', may be provided on the detailed information screen (1430).

[0258] In addition, a 'send all at once' button is provided on the recipe details screen, allowing for a multi-device transfer method without 3D space.

[0259] The processor (280) can display a menu for selecting a home appliance to be transmitted when there are multiple home appliances of the same type in an indoor space.

[0260] Additionally, the processor (280) can transmit the recipe information immediately if the home appliance associated with the recipe information is in the predetermined area, and can display a menu to select whether to transmit if the home appliance associated with the recipe information is in an area other than the predetermined area.

[0261] FIG. 15 illustrates an example of a device selection process. When multiple identical devices are available, a device can be selected, and a recipe can be transmitted to the selected device.

[0262] Referring to FIG. 15 (a), the second device (1512) and the third device (1513) may need to select products to use from the same product family.

[0263] In the selection menu screen (1510), the first device (1511) and the second device (1512) are selected, and the third device (1513) is not selected. In this state, when confirmation is selected, recipe information is transmitted to the first device (1511) and the second device (1512).

[0264] Referring to FIG. 15 (b), the second device (1522) and the third device (1523) may need to select products to use from the same product family.

[0265] In the selection menu screen (1520), the first device (1521) and the third device (1523) are selected, and the second device (1522) is not selected. In this state, when confirmation is selected, recipe information is transmitted to the first device (1521) and the third device (1523).

[0266] If two or more identical products are registered, recipes can only be sent once per product type. In this case, if two or more of the same product are in the On / Standby state on the popup selection menu screen, you must select only one from the product selection popup when sending the recipe (simultaneous sending to two ovens is not possible).

[0267] However, if you send the same recipe card twice, it can be sent separately to Wave Oven 1 and Lightwave Oven 2.

[0268] If only one of the products is turned on / standby, it can be automatically sent to that product without a product selection popup.

[0269] Meanwhile, if there are products that are turned on or standby after a recipe has been sent, additional recipes can be sent. If there are no products available for transmission, product status information can be provided.

[0270] When transmitted to a water purifier, the transmission value of the existing recipe is overwritten when an additional recipe is transmitted even if 1 minute has not elapsed (existing policy maintained).

[0271] Meanwhile, when transmitting recipe information to a lightwave oven or electric range, a transmission cancel button (X) may be provided.

[0272] The button is displayed until the start button is pressed on the product or the product automatically cancels the transmission value.

[0273] When the Cancel button is selected, the transmission to the microwave oven / electric range is canceled.

[0274] If the same recipe is sent to two products and two products are being set up (e.g., if a ramen recipe is sent to lightwave oven 1 and 2), the entire process is canceled when the Cancel button is selected.

[0275] If the request is sent only to the water purifier, a cancel button is not provided. If water is not dispensed, it is automatically cancelled after 1 minute.

[0276] Meanwhile, for certain products, the relevant settings may be automatically cancelled if no separate command is given after the recipe information is transmitted.

[0277] For other specific products, the cancellation does not happen automatically after transmission, and the user must select the cancel button in the appliance management application or turn off the product power.

[0278] Referring to FIG. 16 (a), a three-dimensional space screen (1601) of a top view representing the entire virtual space may be displayed.

[0279] Additionally, a Top View service icon (1610) that allows moving to the dining service screen upon selection may be displayed. When cooking (when a microwave oven or electric range is in use), the Top View service icon (1610) may be displayed as a loading icon (animation).

[0280] Meanwhile, the user can select a specific area (or room).

[0281] Referring to Fig. 16(b), a three-dimensional spatial screen (1602) of an immersive view of a selected specific area may be displayed.

[0282] Additionally, a dining service icon (1620) that allows moving to a dining service screen upon selection may be displayed. The dining service icon (1620) may be provided in a room where one or more dining home appliances (radio oven, electric range, water purifier) ​​are placed.

[0283] When the top view service icon (1610) or dining service icon (1620) is selected, the screen may be switched to the dining service screen.

[0284] FIGS. 17 to 19 are drawings referenced in the description of recipe information transmission on a dining service screen.

[0285] Referring to FIG. 17, the dining service screen (1700) can display all registered dining home appliances (1710, 1720) in one virtual space.

[0286] In addition, the dining service screen (1700) can represent a dining space with graphic elements such as lighting (1740), a dining table (1730), and furniture (1750).

[0287] Multiple recipe cards (1760, 1770, 1780, 1790) may be displayed at the bottom of the dining service screen (1700).

[0288] For recipes where the quantity can be adjusted, such as for one serving, two servings, or a measuring unit, the quantity can be adjusted on the recipe card. A control button (1761) for adjusting the quantity may be displayed on the recipe card (1760) corresponding to the recipe information where the quantity can be adjusted.

[0289] The processor (280) can display a control button (1761) that can adjust the amount on a recipe card (1760) corresponding to the recipe information when the recipe information includes a parameter that allows for adjusting the amount (measuring unit).

[0290] The user can set the amount of the recipe before transmitting the recipe by operating a control button (1761) that can adjust the amount.

[0291] FIG. 20 illustrates various menu styles of a control button (1761) that can adjust the amount.

[0292] FIG. 20(a) illustrates a menu style that allows for quickly setting a small number of items, such as 1 serving, 2 servings, 1 fish, 2 fish, or 1 or 2 items. For example, the user can quickly set 1 or 2 grilled sea snails.

[0293] Figure 20 (b) illustrates a menu style in which the amount of steak can be set in units of 0.1 kg, such as 0.3 to 0.6 kg.

[0294] Fig. 20 (c) illustrates a menu style in which the amount can be adjusted in units of 1 using the + and - buttons, and Fig. 20 (d) illustrates a menu style in which the amount can be adjusted in units of 0.2 using the + and - buttons.

[0295] Referring to FIG. 18, depending on the input of dragging one of the recipe cards (1770) among the multiple recipe cards (1760, 1770, 1780, 1790), the display state of the home appliance object (1710, 1720) associated with the recipe information of the recipe card (1770) may be changed (e.g., highlighted). If the home appliance product corresponding to the home appliance object (1710, 1720) is turned off or offline, it may not be highlighted.

[0296] The changed display state of the home appliance object (1710, 1720) associated with the recipe information can be maintained while dragging the recipe card (1770).

[0297] When the drag is finished, the recipe information of the recipe card (1770) can be transmitted to a home appliance associated with the recipe information.

[0298] When dragging and dropping home appliance objects (1710, 1720) into the product area where they are displayed, the recipe cooking value can be transferred to a product that is in a standby or turned-on state among the associated home appliances.

[0299] Referring to FIG. 19, after the recipe is transmitted, the transmission completion status information may be displayed in the bubble pop-up (1710b, 1720b). Afterwards, the bubble pop-up (1710b) of the oven object (1710) may be updated with information such as remaining time during setup. The bubble pop-up (1720b) ​​of the water purifier object (1720) may be restored to the on state.

[0300] The transmitted recipe can be displayed as the first recipe in the bottom area. Referring to FIG. 19, when the recipe card (1770) is dropped, the recipe card (1770) can be displayed at the very front among the multiple recipe cards (1760, 1770, 1780, 1790). Accordingly, the recipe card (1760) that was previously displayed at the very front can be moved to the back order.

[0301] Meanwhile, cooking status information may be displayed on the transmitted recipe card (1770).

[0302] In addition, a dispensing guidance message (1900) can be provided in the form of a popup immediately after the recipe information is transmitted.

[0303] According to an embodiment, recipe information can be transmitted directly from the top view screen.

[0304] FIGS. 21 to 23 and FIGS. 24 to 25 are drawings referenced for recipe transmission in a top-view screen. The drawings visually represent a scenario in which recipe content is provided in a top-view screen and transmitted to devices placed in multiple rooms. It is possible to provide a method of sending to multiple products at once and a method of selecting and sending to a desired product.

[0305] Referring to FIG. 21, the user interface screen (2100) may include a full area chip (2110) and a dining chip (2120).

[0306] When the dining chip (2120) is selected, the dining home appliance area (2210) can be displayed on the three-dimensional space screen (2200).

[0307] A recipe card (2130) may be displayed on the lower side of the three-dimensional space screen (2200).

[0308] The recipe card (2130) allows you to browse favorite recipes / popular recipes / recommended recipes by scrolling left and right.

[0309] Referring to FIG. 22, one recipe card (2131) among the multiple recipe cards (2131, 2132, 2133) can be transmitted by 'dragging' and 'dropping' it around the dining home appliance area (2210).

[0310] Referring to FIG. 23, you can send a recipe and move to a dining space (virtual space) screen (2300).

[0311] The dining space screen (2300) may include a cooking information item (2310), a recipe item (2320), and a dining space (2330).

[0312] The cooking information item (2310) may display the current cooking status and residual time information.

[0313] The item (2320) for the transmitted recipe may display the transmitted recipe guide information, a cooking cancellation menu button, and a recipe view menu button.

[0314] All home appliance products to which the corresponding recipe has been transmitted can be displayed in the dining space (2330). Referring to FIG. 23, the dining space (2330) can display an oven object (2331) and oven status information (2331a), and a water purifier object (2332) and water purifier status information (2332a).

[0315] Referring to FIG. 24, the user interface screen (2400) includes an entire area chip (2410) and a dining chip (2420), and when the dining chip (2420) is selected, the dining home appliance area (2431) can be displayed in the three-dimensional space screen (2430).

[0316] A cooking start button (2432) may be displayed in the dining home appliance area (2431).

[0317] When the Start Cooking button (2432) is pressed, the bottom sheet (2500) may be exposed.

[0318] Referring to FIG. 25, the bottom sheet (2500) has a plurality of recipe cards (2510) displayed on the upper side, and information (2520, 2530) corresponding to a selected recipe card among the plurality of recipe cards (2510) is displayed on the lower side.

[0319] For example, general information (2520) such as time required, difficulty level, quantity information, and star rating, and detailed information (2530) such as basic ingredients, seasoning ingredients, and cooking method may be displayed.

[0320] After selecting the home appliance (2540) to be transmitted, the cooking value can be transmitted via the ‘transmit recipe’ button (2550).

[0321] After the recipe is transmitted, the screen can be switched to the screen of Fig. 23.

[0322] Meanwhile, cooking menus and cooking status can be shared among members registered in the same home through the home appliance management application.

[0323] FIG. 26 (a) illustrates a screen shown to a member before cooking begins after the recipe is transmitted, and FIG. 26 (b) illustrates a screen shown to a member after cooking begins.

[0324] Referring to Figures 26 (a) and (b), the same screen is shown to all members registered in the home when entering the immersive view.

[0325] The immersive view (2610) may include a three-dimensional spatial screen (2620) representing a selected kitchen area within the entire indoor space and a recipe card (2630).

[0326] Additionally, on the three-dimensional space screen (2620), objects (2621, 2622, 2623) corresponding to home appliances placed in the kitchen space and status information (2621a, 2622a, 2623a) of the home appliances can be displayed.

[0327] After cooking starts, the status information of some home appliances may be updated (2622b, 2623b).

[0328] Before cooking begins, the transmission of a recipe sent by another member can also be canceled. Among the multiple recipe cards (2631, 2632, 2633, 2634), a cancel button (2632a) may be further displayed on the transmitted recipe card (2632).

[0329] After cooking begins, cooking status information (2632b) can be displayed on the recipe card (2632).

[0330] The processor (280) can be configured to allow additional recipe information to be transmitted depending on the type of home appliance, or to allow additional recipe information to be transmitted only after the existing recipe information is canceled.

[0331] FIG. 27 discloses scenarios of operation methods for different situations.

[0332] Referring to FIG. 27, the present disclosure can automatically identify the home appliance to be transmitted based on the number of identical products and the placement location (room) when a user selects a recipe and transmits it to a home appliance, and accordingly, efficiently transmit the recipe.

[0333] Below, examples of recipe transmission methods based on product configuration and layout are explained.

[0334] If you own one of each type of home appliance and they are all placed in one room, the product to be transferred is clear, so if you drag and drop a recipe to transfer it, it is immediately transferred to that home appliance. Meanwhile, home appliance products with transfer values ​​can be highlighted.

[0335] Additional recipes are sent immediately for water purifiers, while for convection ovens and electric ranges, they are sent after canceling the previous recipe.

[0336] If you have one of each type of home appliance and they are placed in multiple rooms, when sending, the product in the current room is highlighted, and that product is automatically selected to send the recipe.

[0337] If you wish to send to a product located in a different room, you can explicitly select the target product through a pop-up screen.

[0338] If you own two or more of the same home appliance products and they are placed in multiple rooms, you can send a recipe in one of the following cases.

[0339] If there are two or more identical home appliance products in the current room, both home appliance products are highlighted when dragging, informing the user that a product can be selected.

[0340] If it is unclear which of the two home appliance products to send to, you can explicitly select the home appliance product to send to through a pop-up screen.

[0341] If there is a previously configured default home appliance product, that product is automatically highlighted and selected, and can be sent immediately without separate selection.

[0342] Additionally, if there are no home appliance products to be transferred in the current room, you can select a product from the pop-up screen to designate the home appliance product to be transferred.

[0343] Additional recipes are transmitted immediately for water purifiers, while transmission may be restricted for convection ovens / electric ranges depending on the previous operation status.

[0344] Water purifiers can be transmitted immediately, and additional transmissions are possible if a convection oven or electric range is on standby.

[0345] In this way, by flexibly identifying the products to be transmitted based on the number and location of products held by the user, and transmitting recipes automatically or semi-automatically according to the user's selection or settings, a more intuitive and efficient recipe transmission user experience can be provided.

[0346] Meanwhile, the processor (280) can set up additional recipes if the same device is already in operation.

[0347] In addition, the processor (280) can recommend recipes that can be cooked using only home appliance products that are not currently being cooked.

[0348] FIG. 28 is a flowchart illustrating a method of operation of an electronic device according to one embodiment of the present disclosure.

[0349] Referring to FIG. 28, the electronic device displays a three-dimensional (3D) spatial screen (S2810). For example, the electronic device may be a mobile terminal (30), and the processor (280) may display a three-dimensional spatial screen including virtual space and home appliance objects on a display (251).

[0350] Additionally, the processor (280) displays a recipe card containing recipe information on the display (251) (S2820). When one or more dining-related home appliances exist in the virtual space, the processor (280) may display the recipe card on the lower side of the virtual space.

[0351] The processor (280) may provide N recipe cards that can be navigated by swiping. At this time, if the number of recipe cards set by the user is less than N, the remaining recipe cards may be provided as recommended recipe cards. The recipe cards set by the user may be placed in a higher order than the recommended recipe cards.

[0352] Step S2810 and Step S2820 can correspond to Steps S410 and S420, respectively.

[0353] The processor (280) controls the recipe information to be transmitted to a predetermined home appliance (S2840) when there is an input to drag and drop the recipe card to a predetermined area in the virtual space (S2830).

[0354] For example, as described with reference to FIGS. 6 to 13, recipe information can be automatically transmitted to an associated home appliance.

[0355] The processor (280) can change the display state of an object corresponding to a home appliance capable of transmitting the recipe information while the recipe card is being dragged.

[0356] Meanwhile, the processor (280) may further display a dining icon (see 1610, 1620 in FIG. 16) when the entire virtual space or a portion of the virtual space is displayed.

[0357] The processor (280) can control the dining home appliances placed in the indoor space to be switched to a dining screen in which all are displayed in one virtual space when the dining icon (1610, 1620) is selected.

[0358] According to an embodiment, the processor (280) can control the dining home appliances placed in the indoor space to switch to a dining screen that is displayed in a single virtual space when the recipe information is transmitted.

[0359] Meanwhile, the processor (280) can control the display of detailed information of the recipe information when there is an input to tap the recipe card (S2835) (S2845).

[0360] For example, as described with reference to Fig. 14, detailed information of the recipe information can be displayed.

[0361] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and it is obvious that various modifications can be made by those skilled in the art without departing from the gist of the present disclosure as claimed in the patent claims.

[0362] The above description may be implemented on computer-readable storage media as one or more computer programs, or as a combination of one or more of the above.

[0363] The functions of the elements disclosed in this document may be implemented using general-purpose processors, special-purpose processors, integrated circuits, ASICs (“Application-Specific Integrated Circuits”), conventional circuits, and / or combinations thereof, or processing circuits. These circuits may be processors configured or programmed to perform the disclosed functions.

[0364] A processor may be regarded as a processing circuit or circuit since it includes transistors and other circuits. In this document, circuits, units, or means may be hardware that performs or is programmed to perform the functions mentioned in the detailed description. Hardware may be hardware disclosed herein or other known hardware, and may be hardware that is programmed or configured to perform the functions mentioned in the detailed description of the specification. Where hardware is a processor that can be regarded as a type of circuit, circuits, means, or units may be a combination of hardware and software, or software used to configure the hardware and / or processor. Additionally, computer storage media may be non-transient computer storage media. For example, it may be executable by a cloud server-based system.

[0365] In addition, the data described in the specification may be computed or executed in various environments, such as cloud server-based systems, on-device systems, and distributed server systems (multiple servers). Processing may be distributed on cloud servers or executed locally on on-device processors, and the results processed in each environment may be stored in non-volatile memory.

[0366] A server (or server system) may be configured to transmit output data for display on a specific electronic device, home appliance, or user display. As another example, the server may configure the output data as a set of computer-readable instructions, such as one or more computer programs. A computer program may be written according to any type of programming language and any programming paradigm. For example, declarative, procedural, assembly, object-oriented, data-oriented, functional, or imperative. The computer program may be written to perform one or more other functions and operate within a computing environment. For example, a physical device, a virtual machine, or across multiple devices. Computer programs may implement the functions described herein, for example, as performed by a system, engine, module, or model.

[0367] A server computing device may be implemented on one or more devices having one or more processors at one or more locations. A user computing device or a server computing device may be communicably connected to one or more storage devices via a network. The storage devices may be a combination of volatile and non-volatile memory and may be located at the same or different physical locations as the computing devices. For example, the storage devices may include various storage drives such as drives, solid-state drives, tape drives, optical storage, memory cards, ROM, RAM, DVDs, CD-ROMs, or readable media within the drives. That is, they may be any type of non-temporary computer storage device capable of storing long-term dedicated data.

[0368] Electronic devices, such as home appliances, and server computing devices may be connected to one or more storage devices via a network. These storage devices may be a combination of volatile and non-volatile memory and may or may not be located in the same physical location as the computing device.

[0369] A server computing device may include one or more processors and memory. The memory stores information that the processor can access and may include data that can be processed, stored, or modified by instructions that can be executed by the processor. Additionally, the memory may consist of volatile and non-volatile memory. The processor may include a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a tensor processing unit (TPU).

[0370] Instructions can be configured to perform specific actions when executed by the processor and can be stored in object code format or interpretable script format. These instructions can be used to implement the system and can be executed on a local or remote processor. Data can be retrieved, stored, or modified according to the instructions and can be organized in database, JSON, YAML, or XML formats.

[0371] Electronic devices, including home appliances and hubs, can be configured similarly to server computing devices. Electronic devices, including home appliances and hubs, may include user input and output devices in addition to processors, memory, instructions, and data. Server computing devices can transmit data to said electronic devices, and said electronic devices can display a portion of the received data through a display. Furthermore, data transmission and communication between said server computing devices and electronic devices are possible through networks such as Bluetooth, Wi-Fi, wired, and wireless networks, and direct and indirect communication between computing devices is possible, supporting various protocols and connection methods.

Claims

1. A display that displays a three-dimensional spatial screen including a virtual space representing an indoor space and an object corresponding to a home appliance placed in the indoor space; and, A processor that displays a recipe card containing recipe information on the above display; The above processor is, If there is an input to drag and drop the above recipe card to a predetermined area in the above virtual space, the recipe information is controlled to be transmitted to a predetermined home appliance, and An electronic device that controls the display of detailed information of the recipe when there is an input to tap the recipe card.

2. In Paragraph 1, The above processor is, An electronic device that displays the recipe card at the bottom of the virtual space when one or more dining-related home appliances exist in the virtual space.

3. In Paragraph 1, The above processor is, Provides N recipe cards that can be explored by swiping, and An electronic device that provides the remaining recipe cards as recommended recipe cards when the number of recipe cards set by the user is less than the above N.

4. In Paragraph 3, An electronic device in which the recipe card set by the user is placed in a higher order than the recommended recipe card.

5. In Paragraph 1, The above processor is, An electronic device that changes the display state of an object corresponding to a home appliance capable of transmitting the recipe information while the above recipe card is being dragged.

6. In Paragraph 1, The above processor is, While the entire virtual space or a portion of the virtual space is displayed, additional dining icons are displayed, and An electronic device that controls the switching to a dining screen in which all dining home appliances placed in the indoor space are displayed in a single virtual space when the above dining icon is selected.

7. In Paragraph 1, The above processor is, An electronic device that controls the switching of all dining home appliances placed in the indoor space to a dining screen displayed in a single virtual space when the above recipe information is transmitted.

8. A step of displaying a three-dimensional space screen including a virtual space representing an indoor space and an object corresponding to a home appliance placed in the indoor space; A step of displaying a recipe card containing recipe information; If there is an input in which the recipe card is dragged and dropped to a predetermined area in the virtual space, the recipe information is transmitted to a predetermined home appliance; and, A method of operating an electronic device comprising the step of displaying detailed information of the recipe information when there is an input of tapping the recipe card.

9. A storage medium storing a computer-readable program for performing a method of operating an electronic device, The method of operating the above electronic device is, A step of displaying a three-dimensional space screen including a virtual space representing an indoor space and an object corresponding to a home appliance placed in the indoor space, Step of displaying a recipe card containing recipe information, If there is an input in which the recipe card is dragged and dropped to a predetermined area in the virtual space, the recipe information is transmitted to a predetermined home appliance, and, A storage medium comprising the step of displaying detailed information of the recipe information when there is an input of tapping the recipe card.

Images