Refrigerator comprising display and control method therefor

Abstract

A refrigerator is provided. The refrigerator comprises: a storage chamber accommodating a stored object; a camera photographing at least a portion of the storage chamber; an input interface receiving a user input; a display; at least one processor; and a memory storing a plurality of instructions. The refrigerator recognizes an object stored in the storage chamber from an input image captured by the camera, converts a two-dimensional (2D) object image of the recognized object obtained from the input image into a three-dimensional (3D) object image, arranges the converted 3D object image on a 3D interior image, and changes at least one of a display area or a display direction of the 3D interior image when a first user input for manipulating the 3D interior image is received while the 3D interior image is displayed on the display.

Description

Refrigerator including a display and control method thereof

[0001] Embodiments of the present disclosure relate to a refrigerator including a display, a refrigerator control method, and a recording medium storing a computer program for performing the refrigerator control method.

[0002] Refrigerators accommodate and store multiple food items in their storage compartments. These compartments include multiple shelves, drawers, and partitions to separate the internal space. Additionally, the compartments have a specific depth to provide a storage capacity. However, when storing multiple food items, some may end up recessed into the interior; in such cases, the food located further inside is not easily visible from the outside, making it difficult for the user to locate the items within. Furthermore, this difficulty in finding food within the compartment leads to problems such as food not being consumed before their expiration date, resulting in spoilage or a decline in freshness.

[0003] According to one aspect of an embodiment of the present disclosure, a refrigerator is provided. The refrigerator includes a storage compartment for receiving an object that has been received, a camera that captures at least a portion of the storage compartment, an input interface that receives user input, a display, at least one processor, and a memory that stores a plurality of instructions. When the plurality of instructions are executed selectively or collectively by at least one processor, the refrigerator recognizes an object that has been received into the storage compartment from an input image captured by the camera, converts a 2D (2-Dimension) object image of the recognized object obtained from the input image into a 3D (3-Dimension) object image, places the converted 3D object image on a 3D storage image, and when a first user input is received to manipulate the 3D storage image while the 3D storage image is displayed on the display, changes at least one of the display area or display direction of the 3D storage image.

[0004] Additionally, according to one aspect of an embodiment of the present disclosure, a refrigerator control method is provided. The refrigerator control method includes the steps of: recognizing an object that has entered the storage room from an input image captured by a camera placed in the storage room of the refrigerator; converting a 2D object image of the recognized object obtained from the input image into a 3D object image; placing the converted 3D object image on a 3D interior image; and, when a first user input that manipulates the 3D interior image is received while the 3D interior image is displayed, changing at least one of the display area or display direction of the 3D interior image.

[0005] In addition, according to one aspect of one embodiment of the present disclosure, a computer-readable recording medium is provided on which a program for performing a refrigerator control method on a computer is recorded.

[0006] FIG. 1 is a drawing showing a refrigerator and a 3D interior image according to one embodiment of the present disclosure.

[0007] FIG. 2 is a block diagram showing the structure of a refrigerator according to one embodiment of the present disclosure.

[0008] FIG. 3 is a flowchart illustrating a refrigerator control method according to one embodiment of the present disclosure.

[0009] FIG. 4 is a drawing showing the structure of a refrigerator (100) according to one embodiment of the present disclosure.

[0010] FIG. 5 is a drawing showing an upper perspective view of a refrigerator according to one embodiment of the present disclosure.

[0011] FIG. 6 is a diagram illustrating the process of generating a 2D object image from an input image according to one embodiment of the present disclosure.

[0012] FIG. 7 is a diagram illustrating a process of generating a 2D object image from an input image including a plurality of objects according to one embodiment of the present disclosure.

[0013] FIG. 8 is a diagram illustrating a process of generating a 3D object image from a 2D object image according to one embodiment of the present disclosure.

[0014] FIG. 9 is a diagram illustrating the process of identifying location information of an object within a storage room according to one embodiment of the present disclosure.

[0015] FIG. 10 is a drawing showing a plurality of storage compartment areas of a storage compartment of a refrigerator according to one embodiment of the present disclosure.

[0016] FIG. 11 is a diagram illustrating a process of identifying location information of an object entering a storage room according to one embodiment of the present disclosure.

[0017] FIG. 12 is a diagram illustrating the process of estimating location information where an object is received in a storage room according to one embodiment of the present disclosure.

[0018] FIG. 13 is a diagram showing the rotation of a 3D internal image in 3D space in a 3D vision GUI according to one embodiment of the present disclosure.

[0019] FIG. 14 is a diagram illustrating the process of zooming in on a display area in a 3D vision GUI according to one embodiment of the present disclosure.

[0020] FIG. 15 is a diagram illustrating a process of changing the display area or display direction of a 3D internal image in response to a first user input according to one embodiment of the present disclosure.

[0021] FIG. 16 is a drawing showing the distance between a plurality of objects in a 3D vision GUI increased and displayed according to one embodiment of the present disclosure.

[0022] FIG. 17 is a drawing showing the distance between a plurality of objects in a 3D vision GUI increased and displayed according to one embodiment of the present disclosure.

[0023] FIG. 18 is a drawing showing the distance between a plurality of objects in a 3D vision GUI increased and displayed according to one embodiment of the present disclosure.

[0024] FIG. 19 is a diagram illustrating the process of determining the display area and display direction of a 3D internal image according to one embodiment of the present disclosure.

[0025] FIG. 20 is a block diagram showing the structure of a refrigerator according to one embodiment of the present disclosure.

[0026] FIG. 21 is a diagram illustrating a process of obtaining user identification information according to one embodiment of the present disclosure.

[0027] FIG. 22 is a diagram illustrating a process for obtaining storage object information and an example of storage object information according to one embodiment of the present disclosure.

[0028] FIG. 23 is a flowchart illustrating the process of outputting location information of an object using voice recognition according to one embodiment of the present disclosure.

[0029] FIG. 24 is a diagram illustrating the process of outputting location information of an object using voice recognition according to one embodiment of the present disclosure.

[0030] FIG. 25 is a flowchart illustrating a process for obtaining keywords for unidentifiable objects in an input image according to one embodiment of the present disclosure.

[0031] FIG. 26 is a diagram illustrating a process of generating a food image from a keyword according to one embodiment of the present disclosure.

[0032] FIG. 27 is a drawing showing a food image displayed on an object when the object is not visible, according to one embodiment of the present disclosure.

[0033] FIG. 28 is a diagram illustrating the process of obtaining keywords for delivery food according to one embodiment of the present disclosure.

[0034] FIG. 29 is a drawing showing a refrigerator, an external device, and a server according to one embodiment of the present disclosure.

[0035] The various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments.

[0036] In relation to the description of the drawings, similar reference numerals may be used for similar or related components.

[0037] The singular form of the noun corresponding to the item may include one or multiple items, unless the relevant context clearly indicates otherwise.

[0038] In the present disclosure, each of the phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0039] The term "and / or" includes a combination of multiple related described components or any of the multiple related described components.

[0040] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0041] Additionally, terms such as 'front,' 'rear,' 'top,' 'bottom,' 'side,' 'left,' 'right,' 'top,' and 'bottom' used in this disclosure are defined based on the drawings, and the shape and location of each component are not limited by these terms.

[0042] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this disclosure, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0043] When it is said that a component is "connected," "combined," "supported," or "in contact" with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0044] When it is said that a component is located "on" another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0045] A refrigerator according to one embodiment may include a main body.

[0046] The "main body" may include an inner body, an outer body positioned on the outside of the inner body, and an insulating material provided between the inner body and the outer body.

[0047] The "inner body" may include at least one of a case, plate, panel, or liner forming a storage chamber. The inner body may be formed as a single body or may be formed by assembling multiple plates. The "outer body" may form the exterior of the main body and may be coupled to the outer side of the inner body so that an insulating material is disposed between the inner body and the outer body.

[0048] The "insulating material" can insulate the interior and exterior of the storage room so that the temperature inside the storage room is maintained at a set appropriate temperature without being affected by the external environment. According to one embodiment, the insulating material may include a foamed insulating material. The foamed insulating material can be formed by injecting and foaming urethane foam, which is a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0049] According to one embodiment, the insulation material may additionally include a vacuum insulation material in addition to a foam insulation material, or the insulation material may consist solely of a vacuum insulation material instead of a foam insulation material. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation material or vacuum insulation material described above and may include various materials that can be used for insulation.

[0050] The "storage room" may include a space defined by an internal structure. The storage room may further include an internal structure defining a space corresponding to the storage room. Various items such as food, medicine, and cosmetics may be stored in the storage room, and the storage room may be formed so that at least one side is open to allow for the retrieval and retrieval of items.

[0051] A refrigerator may include one or more storage compartments. When two or more storage compartments are formed in a refrigerator, each storage compartment may have a different use and may be maintained at a different temperature. To this end, each storage compartment may be partitioned from one another by a partition containing insulation.

[0052] The storage room may be provided to be maintained within an appropriate temperature range according to its intended use and may include a "refrigeration room," "freezing room," or "variable temperature room" distinguished according to its intended use and / or temperature range. The refrigerator room may be maintained at a temperature suitable for refrigerated storage of goods, and the freezer room may be maintained at a temperature suitable for frozen storage of goods. "Refrigeration" may mean cooling goods to a temperature that does not freeze them; for example, the refrigerator room may be maintained within a range of 0 degrees Celsius to 7 degrees Celsius. "Freezing" may mean cooling goods to freeze them or to maintain them in a frozen state; for example, the freezer room may be maintained within a range of -20 degrees Celsius to -1 degree Celsius. The variable temperature room may be used as either a refrigerator room or a freezer room, with or without the user's choice.

[0053] Storage rooms may be referred to by various names, such as "vegetable room," "fresh room," "cooling room," and "ice-making room," in addition to terms like "refrigeration room," "freezing room," and "variable temperature room." The terms "refrigeration room," "freezing room," and "variable temperature room" used below should be understood as encompassing storage rooms with corresponding uses and temperature ranges.

[0054] According to one embodiment, the refrigerator may include at least one door configured to open and close one side of the storage compartment. The door may be provided to open and close each of one or more storage compartments, or a single door may be provided to open and close multiple storage compartments. The door may be installed to be rotatable or sliding on the front of the main body.

[0055] The “door” may be configured to seal the storage room when the door is closed. The door may include insulation material, similar to the main body, to insulate the storage room when the door is closed.

[0056] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the rear of the door and facing the storage room, an upper cap, a lower cap, and a door insulation material provided inside the same.

[0057] A gasket may be provided on the edge of the door inner panel to seal the storage compartment by adhering to the front of the main body when the door is closed. The door inner panel may include a dyke that protrudes rearward to allow a door basket for storing items to be mounted.

[0058] According to one embodiment, the door may include a door body and a front panel detachably coupled to the front side of the door body and forming the front of the door. The door body may include a door outer panel forming the front of the door body, a door inner panel forming the rear of the door body and facing the storage compartment, an upper cap, a lower cap, and a door insulation material provided inside them.

[0059] Refrigerators can be classified into French Door Type, Side-by-side Type, BMF (Bottom Mounted Freezer), TMF (Top Mounted Freezer), or 1-door refrigerators depending on the arrangement of the door and storage compartment.

[0060] According to one embodiment, the refrigerator may include a cold air supply device arranged to supply cold air to the storage compartment.

[0061] The "cold air supply device" may include a machine, apparatus, electronic device, and / or a system combining these that can generate cold air and guide cold air to cool a storage room.

[0062] According to one embodiment, a cold supply device can generate cold air through a refrigeration cycle that includes the processes of compression, condensation, expansion, and evaporation of a refrigerant. To this end, the cold supply device may include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. According to one embodiment, the cold supply device may include a semiconductor such as a thermoelectric element. The thermoelectric element can cool a storage chamber through heat generation and cooling action via the Peltier effect.

[0063] According to one embodiment, the refrigerator may include a machine room arranged to accommodate at least some parts belonging to a cold air supply device.

[0064] The "machine room" may be configured to be partitioned and insulated from the storage room to prevent heat generated from components placed in the machine room from being transferred to the storage room. The interior of the machine room may be configured to communicate with the exterior of the main body to dissipate heat from components placed inside the machine room.

[0065] According to one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without opening the door.

[0066] According to one embodiment, the refrigerator may include an ice-making device configured to generate ice. The ice-making device may include an ice-making tray that stores water, an ice-removing device that separates ice from the ice-making tray, and an ice bucket that stores the ice generated from the ice-making tray.

[0067] According to one embodiment, the refrigerator may include a control unit for controlling the refrigerator.

[0068] The "control unit" may include a memory that stores or remembers a program and / or data for controlling a refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc., according to the program and / or data stored in the memory.

[0069] The memory stores or records various information, data, commands, programs, etc., necessary for the operation of the refrigerator. The memory can store temporary data generated while generating control signals to control the components included in the refrigerator. The memory may include at least one of volatile memory or non-volatile memory, or a combination thereof.

[0070] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operation of an artificial intelligence model. Additionally, the processor may include a central processing unit, a graphics processing unit (GPU), etc. The processor can generate control signals to control the operation of the cold air supply unit. For example, the processor can receive temperature information of the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cold air supply unit based on the temperature information of the storage compartment.

[0071] Additionally, the processor can process user input of the user interface and control the operation of the user interface according to programs and / or data stored in memory. The user interface may be provided using an input interface and an output interface. The processor can receive user input from the user interface. Additionally, the processor can transmit display control signals and image data to the user interface to display an image on the user interface in response to the user input.

[0072] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one sub-processor. The memory may include one or more memory units.

[0073] According to one embodiment, the refrigerator may include a processor and memory that control all components included in the refrigerator, and may include a plurality of processors and a plurality of memories that individually control the components of the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cold air supply device according to the output of a temperature sensor. Additionally, the refrigerator may separately provide a processor and memory that control the operation of a user interface according to user input.

[0074] The communication module can communicate with external devices, such as servers, mobile devices, and other home appliances, through nearby Access Points (APs). The Access Point (AP) can connect the Local Area Network (LAN) to which the refrigerator or user device is connected to the Wide Area Network (WAN) to which the server is connected. The refrigerator or user device can be connected to the server through the Wide Area Network (WAN).

[0075] The input interface may include keys, touchscreens, microphones, etc. The input interface may receive user input and transmit it to the processor.

[0076] The output interface may include a display, a speaker, etc. The output interface can output various notifications, messages, information, etc. generated by the processor.

[0077] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0078] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0079] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0080] Refrigerators according to various embodiments will be described in detail below with reference to the attached drawings.

[0081] FIG. 1 is a drawing showing a refrigerator and a 3D interior image according to one embodiment of the present disclosure.

[0082] According to one embodiment of the present disclosure, a refrigerator (100) provides a GUI (Graphic User Interface) view that shows the interior of a storage room (110) in 3D. The refrigerator (100) includes a camera (120) inside the storage room (110). The camera (120) is positioned on the upper surface of the storage room (110) to photograph the front of the opening of the storage room (110) when the door of the refrigerator (100) is opened. When the door is opened, the camera (120) generates an input image of the front of the storage room (110).

[0083] In the present disclosure, the object (112) entering the storage room (110) may include food, medicine, or cosmetics. The object (112) may be entered into the storage room in a state contained in a bag or container. The object (112) may include various types of food, such as ingredients, cooked food, ready-made products, and delivery food.

[0084] The refrigerator (100) recognizes an object (112) that a user (114) places into the storage room (120) from an input image captured by a camera (120). The refrigerator (100) converts the object (112) recognized from the input image into a 3D (dimension) image. In this disclosure, a 3D image generated by converting an area corresponding to the object (112) in the input image from a 2D image into a 3D image is referred to as a 3D object image. The refrigerator (100) places the 3D object image on a 3D interior image that implements the interior of the storage room (110) as a 3D image. The refrigerator (100) displays the 3D interior image with the 3D object image placed on a display (130). The refrigerator (100) provides a 3D vision GUI that displays the 3D interior image through the display (130). The 3D vision GUI places a 3D object image on a 3D internal image to show the interior of a virtually implemented storage room (110).

[0085] When the refrigerator (100) receives a first user input that operates a 3D vision GUI, it may change at least one of the display area or display direction of the 3D interior image displayed in the 3D vision GUI according to the first user input. For example, the display (130) is a touchscreen, and the first user input may correspond to a touch and drag input (150) entered through the touchscreen. According to one embodiment of the present disclosure, when the refrigerator (100) receives a user's touch and drag input (150), it may change the display area and display direction of the 3D interior image according to the drag direction of the touch and drag input (150). The display area refers to the area displayed in the 3D interior image on the 3D vision GUI. The display area may be defined, for example, by the width of the displayed area. Additionally, the display area may be defined by the position of the displayed area in the 3D interior image. The display direction refers to the direction in which the 3D interior image is displayed in the 3D vision GUI. The display direction may include, for example, a top view, a front view, or an oblique view.

[0086] According to one embodiment of the present disclosure, the refrigerator (100) provides a 3D vision GUI, thereby enabling the user to conveniently view the inside of the storage room (110). Additionally, according to one embodiment of the present disclosure, the refrigerator (100) provides a 3D vision GUI, thereby enabling the user to easily find and manage objects (112) inside the storage room (110).

[0087] FIG. 2 is a block diagram showing the structure of a refrigerator according to one embodiment of the present disclosure.

[0088] According to one embodiment of the present disclosure, the refrigerator (100) may include a storage compartment (110), a camera (120), a display (130), a processor (210), an input interface (212), and a memory (214).

[0089] The storage room (110) accommodates incoming objects and corresponds to a storage space with one side open. The storage room (110) is provided in the main body of the refrigerator (100). The storage room (110) can be opened or closed by a door attached to the main body. The storage room (110) can be implemented with a material and structure capable of thermal insulation. The storage room (110) can be implemented in various types depending on the target temperature. For example, the storage room (110) may include a refrigerator room, a freezer room, a kimchi storage room, a wine storage room, or an aging room. The refrigerator (100) may include a plurality of storage rooms (110). The plurality of storage rooms may be opened and closed by different doors. For example, the refrigerator (100) may include one refrigerator room and two freezer rooms. Also, for example, one refrigerator room may be opened and closed by two doors.

[0090] The storage room (110) may be divided into multiple storage room areas by shelves, drawers, partitions, or door baskets, etc. The multiple storage room areas may be defined by shelves, left, right, drawers, etc. For example, the multiple storage room areas may be defined as the first shelf on the left, the second shelf on the left, the first shelf on the right, the second shelf on the right, the left vegetable room, the right vegetable room, or the aging room.

[0091] A camera (120) may be placed inside the storage room (110). The camera (120) may be placed to photograph the front of the storage room (110). When the door is opened, the camera (120) generates an input image of the front of the storage room (110). When the event of the door opening is detected, the camera (120) may start photographing the front of the storage room (110). The camera (120) may record a video of the front of the storage room (110) at a predetermined frame rate. The input image may correspond to a video or a still image captured while the door is opening.

[0092] The camera (120) may include a lens, an aperture, and an image sensor. The camera (120) outputs a signal detected from the image sensor to a processor (210), and the processor (210) can generate an input image from the detected signal received from the camera (120).

[0093] A display (130) may be placed on the door of the refrigerator (100). The display (130) may display the status of the refrigerator (100), GUI, or related information. The display (130) may be implemented as a touchscreen.

[0094] The input interface (212) can receive various inputs from a user. The input interface (212) may include, for example, a touchscreen, a key, a button, a touchpad, or a microphone. The input interface (212) can receive a first user input that operates a 3D vision GUI.

[0095] The first user input can correspond to a touch gesture input that manipulates the 3D vision GUI. The touch gesture input can be input through a touchscreen.

[0096] Additionally, the input interface (212) can receive user voice through a microphone. The refrigerator (100) can recognize user input controlling the refrigerator (100) by recognizing the user voice. Additionally, for example, the refrigerator (100) can recognize a first user input operating a 3D vision GUI by recognizing the user voice.

[0097] The processor (210) controls the overall operation of the refrigerator (100). The processor (210) may be implemented as one or more processors. The processor (210) may perform a predetermined operation by executing instructions or commands stored in memory (214). Additionally, the processor (210) controls the operation of components provided in the refrigerator (100). The processor (210) may include a CPU (Central Processing Unit), a microprocessor, etc. The processor (210) may include at least one processing circuitry. When the processor (210) selectively or collectively executes instructions stored in memory (214), it may control the refrigerator (100) to perform an operation according to the instructions.

[0098] The memory (214) stores various information, data, commands, programs, etc., necessary for the operation of the refrigerator (100). The memory (214) may include at least one of volatile memory or non-volatile memory, or a combination thereof. The memory (214) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Additionally, the memory (214) may correspond to a web storage or cloud server that performs storage functions over the internet.

[0099] The processor (210) recognizes an object (112) that has entered the storage room (110) from an input image captured by a camera (120). The processor (210) can recognize an object that a user holds in their hand and enters the storage room (110) from a frame of the input image. The processor (210) can recognize an object (112) that has entered the storage room (110) from the input image using an object recognition algorithm. According to one embodiment of the present disclosure, the processor (210) can recognize an object (112) from the input image using a learned object recognition artificial intelligence model. The object recognition artificial intelligence model may be an artificial intelligence model machine-learned from a plurality of learning data that takes an input image captured by a camera (120) placed on the upper surface of the storage room (110) as input and a recognized object (112) as output. According to one embodiment of the present disclosure, the object recognition artificial intelligence model may be executed by the processor (210). In addition, according to one embodiment of the present disclosure, the object recognition artificial intelligence model can be executed by an external server.

[0100] Additionally, the processor (210) obtains an area corresponding to an object (112) recognized from the input image from the input image. The area corresponding to the object (112) obtained from the 2D input image corresponds to a 2D object image. For example, the processor (210) can generate a 2D object image by cropping an area corresponding to the object (112) from the input image. The processor (210) can convert the 2D object image into a 3D object image. According to one embodiment of the present disclosure, the processor (210) can generate a 3D object image from a 2D object image using a 3D object conversion artificial intelligence model. The 3D object conversion artificial intelligence model may be an artificial intelligence model machine-learned using a plurality of training data that takes a 2D object image obtained from an input image captured by a camera (130) placed on the upper surface of the storage room (110) as input and a converted 3D object image as output. According to one embodiment of the present disclosure, the 3D object conversion artificial intelligence model may be executed by the processor (210). In addition, according to one embodiment of the present disclosure, the 3D object transformation artificial intelligence model can be executed by an external server.

[0101] Additionally, the processor (210) can place a 3D object image on the 3D interior image. The 3D interior image is an image that represents the storage room (110) in 3D. For each storage room (110) included in the refrigerator (100), a 3D interior image may be stored in advance in memory (214). The 3D interior image may correspond to the refrigerator (100) model name. The processor (210) can identify the location where an object (112) is placed in the storage room (110) from the input image. The operation of identifying the location of the object (112) is described in detail below. Based on the location information of the object (112) within the storage room (110), the processor (210) places the 3D object image on the 3D interior image. For example, if the location information of an object (112) corresponds to the second shelf from the left, the processor (210) can place the 3D object image at a position corresponding to the second shelf from the left in the 3D internal image. According to one embodiment of the present disclosure, the 3D internal image and the 3D object image may correspond to different image layers. The processor (210) can display the 3D internal image with the 3D object image placed on the display (130). The processor (210) can display a 3D vision GUI including the 3D internal image with the 3D object image placed. According to one embodiment of the present disclosure, the 3D vision GUI can display the 3D internal image with the 3D internal image placed.

[0102] Additionally, when the processor (210) receives a first user input that manipulates the 3D vision GUI, it can change the display area or display direction of the 3D internal image.

[0103] The processor (210) can zoom in or zoom out on the 3D interior image to change the display area. Additionally, the processor (210) can change the position of some areas displayed in the 3D interior image while zooming in on the 3D interior image to change the display area. For example, the processor (210) can change the display area by displaying an area corresponding to the second shelf on the left while transitioning from a first state in which the entire area of ​​the storage room (110) is displayed to a zoom-in state in which the 3D interior image is enlarged. Additionally, the processor (210) can change the display area by transitioning from a state in which the area corresponding to the second shelf on the left is displayed in the zoom-in state to a state in which the area corresponding to the second shelf on the right is displayed in the zoom-in state.

[0104] Additionally, the processor (210) can change the direction in which the 3D internal image is displayed in order to change the display direction. For example, the processor (210) can switch from a top view to a diagonal view in order to change the display direction.

[0105] According to one embodiment of the present disclosure, when a touch and drag input is detected within an area where a 3D internal image is displayed while a 3D vision GUI is displayed on a touchscreen, the processor (210) can rotate the display direction of the 3D internal image according to the drag input of the touch and drag input.

[0106] FIG. 3 is a flowchart illustrating a refrigerator control method according to one embodiment of the present disclosure.

[0107] A refrigerator control method according to one embodiment of the present disclosure can be performed by a refrigerator (100) according to one embodiment of the present disclosure.

[0108] Referring to FIG. 3, in step S302, the refrigerator (100) recognizes an object (112) that has been brought into the storage room (110) from an input image captured by a camera (120). The refrigerator (100) can generate a 2D object image of the object (112) recognized from the input image. The refrigerator (100) can generate a 2D object image using an object recognition artificial intelligence model.

[0109] Next, the refrigerator (100) can generate a 3D object image by converting a 2D object image into a 3D object image in step S304. The refrigerator (100) can generate a 3D object image using a 3D object conversion artificial intelligence model.

[0110] Next, in step S306, the refrigerator (100) places a 3D object image onto a 3D interior image. The refrigerator (100) can identify the location where the object (112) is placed in the storage room (110) from the input image. Based on the location information of the object (112) within the storage room (110), the refrigerator (100) places the 3D object image onto the 3D interior image. The refrigerator (100) can display a 3D vision GUI that includes the 3D interior image on which the 3D object image is placed.

[0111] Next, when the refrigerator (100) receives a first user input that operates the 3D vision GUI, it can change the display area or display direction of the 3D interior image. To change the display area, the refrigerator (100) can zoom in or zoom out the 3D interior image. Additionally, to change the display area, the refrigerator (100) can change the position of some areas displayed in the 3D interior image while zooming in to display the 3D interior image. Additionally, to change the display direction, the refrigerator (100) can switch the direction in which the 3D interior image is displayed. Furthermore, according to one embodiment of the present disclosure, when the refrigerator (100) detects a touch-and-drag input within the area where the 3D interior image is displayed while the 3D vision GUI is displayed on the touchscreen, it can rotate the display direction of the 3D interior image according to the drag input of the touch-and-drag input.

[0112] FIG. 4 is a drawing showing the structure of a refrigerator (100) according to one embodiment of the present disclosure.

[0113] According to one embodiment of the present disclosure, a refrigerator (100) has an open state (420) and a closed state (422). The refrigerator (100) may include a plurality of doors (410a, 410b, 410c, and 410d). In the present disclosure, the plurality of doors (410a, 410b, 410c, and 410d) are collectively referred to as identification number 410. The storage compartment (110) of the refrigerator (100) can be switched to an open state (420) or a closed state (422) by opening or closing the door (410).

[0114] The camera (120) can be placed on the upper surface of the storage room (110). The camera (120) can generate an input image by capturing the front of the storage room (120) in an open state (420). The camera (120) is deactivated in a closed state (422) and is activated in an open state (420) to perform a shooting operation and generate an input image.

[0115] The display (130) may be placed on at least one door (410). For example, the display (130) may be placed on an upper door (410a or 410b). The display (130) may be a touchscreen. Additionally, for example, the display (130) may be installed on each of a plurality of doors (410).

[0116] The refrigerator (100) can display a GUI on the display (130) that displays information related to the refrigerator (100). For example, the refrigerator (100) can display a GUI on the display (130) that provides information about incoming food, medicine, cosmetics, etc., event information, alarm information, or recipe information. The refrigerator (100) can collect and store information about incoming objects such as food, medicine, or cosmetics, and continuously update it. The refrigerator (100) can display information about the objects through the display (130).

[0117] FIG. 5 is a drawing showing an upper perspective view of a refrigerator according to one embodiment of the present disclosure.

[0118] The refrigerator (100) can capture an area corresponding to a predetermined FOV (Field of View, 510) by a camera (120). Depending on the degree to which the door (410) is opened in the open state (420) of the refrigerator (100), the input image captured by the camera (120) may vary. For example, in a first state (520) in which the door (410) is opened such that the angle with the main body (530) is 90 degrees or more, the input image captured by the camera (120) may correspond to an image in which the door basket of the door (410) is not captured, and the area in front of the storage room (110) where the door (410) is opened is captured. In a second state (522) in which the door (410) is opened such that the angle with the main body (530) is less than 90 degrees, an input image including the door basket is captured. According to one embodiment of the present disclosure, since an input image is used in which a user holds an object (112) in their hand and puts it into a storage room (110), the refrigerator (100) can recognize the object (112) being put into the storage room (110) from the input image by using the input image of the first state (520).

[0119] FIG. 6 is a diagram illustrating the process of generating a 2D object image from an input image according to one embodiment of the present disclosure.

[0120] According to one embodiment of the present disclosure, a refrigerator (100) generates a 2D object image from an input image (610). The input image (610) illustrated in FIG. 6 is an image corresponding to the whole or part of an image captured by a camera (120). The refrigerator (100) recognizes an object region (620) corresponding to an object (112) from the input image (610). According to one embodiment of the present disclosure, the refrigerator (100) can separate the object region (620) from the background region. The refrigerator (100) can mask the background region.

[0121] According to one embodiment of the present disclosure, a refrigerator (100) can generate an image of an object region (620) with the background region removed. The refrigerator (100) can input the image of the object region (620) into an object restoration artificial intelligence model and obtain a 2D object image (630) generated by the object restoration artificial intelligence model.

[0122] FIG. 7 is a diagram illustrating a process of generating a 2D object image from an input image including a plurality of objects according to one embodiment of the present disclosure.

[0123] According to one embodiment of the present disclosure, a refrigerator (100) can generate a plurality of 2D object images from an input image (710) containing a plurality of objects. The plurality of objects (112a, 112b) may correspond to objects of the same type or may correspond to objects of different types. The refrigerator (100) can recognize a plurality of objects (112a, 112b) from the input image (710). The refrigerator (100) can recognize a plurality of object regions (720a, 720b) ​​corresponding to each of the plurality of objects (112a, 112b). The refrigerator (100) can divide a first object region (720a), a second object region (720b), and a background region. The refrigerator (100) can remove the background region and generate an image corresponding to the first object region (720a) and an image corresponding to the second object region (720b). The refrigerator (100) inputs an image corresponding to a first object region (720a) and an image corresponding to a second object region (720b) ​​into an object restoration artificial intelligence model, respectively, and can obtain a first 2D object image (730a) and a second 2D object image (730b) generated by the object restoration artificial intelligence model.

[0124] The refrigerator (100) can accurately recognize the number of objects (112a, 112b) being received or shipped out by recognizing each of the multiple objects (112a, 112b) of the same type. In addition, the refrigerator (100) has the effect of being able to insert each of the multiple objects (112a, 112b) into the correct position with correct placement by separately generating 2D object images (730a, 730b) for each of the multiple objects (112a, 112b) of the same type when placing each of the multiple objects (112a, 112b) in the 3D internal image.

[0125] FIG. 8 is a diagram illustrating a process of generating a 3D object image from a 2D object image according to one embodiment of the present disclosure.

[0126] According to one embodiment of the present disclosure, a refrigerator (100) generates a 3D object image (814) from a 2D object image (812) using a 3D object transformation artificial intelligence model (810). The 3D object transformation artificial intelligence model (810) is an artificial intelligence model that is machine-learned using a plurality of training data that takes a 2D object image (812) as input and a 3D object image (814) as output.

[0127] According to one embodiment of the present disclosure, a 3D object transformation artificial intelligence model (810) can be performed by a processor (210) of a refrigerator (100).

[0128] According to one embodiment of the present disclosure, a 3D object conversion artificial intelligence model (810) may be performed by an external server. A refrigerator (100) transmits a 2D object image (812) to a server, and the server may convert the received 2D object image (812) into a 3D object image (814) using the 3D object conversion artificial intelligence model (810). The server may transmit the 3D object image (814) to the refrigerator (100).

[0129] FIG. 9 is a diagram illustrating the process of identifying location information of an object within a storage room according to one embodiment of the present disclosure.

[0130] According to one embodiment of the present disclosure, a refrigerator (100) identifies the location information of the object (112) finally placed in the storage room (110) by using an input image that captures the appearance of the object (112) being placed in the storage room (110). Since the camera (120) placed in the storage room (110) does not photograph the interior of the storage room (110) but photographs the front of the refrigerator from the storage room (110), the refrigerator (100) can recognize the location information of the object (112) finally placed inside the storage room (110) from the input image. Accordingly, according to one embodiment of the present disclosure, the refrigerator (100) can identify the location information where the object (112) is finally placed inside the storage room (110) by using the movement of the hand and arm of the user (114) who has grasped the object (112) in the input image, the gaze, the movement path, the change in size of the object (112), or the motion vector of the object, the motion vector of the arm, etc.

[0131] In step 902, the refrigerator (100) uses a camera (120) to capture the user (114) holding the object (112) in their hand and placing the object (112) into the storage room (110) to generate an input video. The refrigerator (100) can record the scene of placing the object (112) into the storage room as a first video. From the first video, the refrigerator (100) can recognize the movement of the user (114)'s hand and arm, gaze, movement path, change in the size of the object (112), or motion vectors.

[0132] In step 904, the refrigerator (100) can identify the location where the object (112) is finally placed in the storage room (110). The refrigerator (100) can determine that the object (112) is finally placed in the storage room (110) if it recognizes from the input video that the user (114) puts the object (112) into the storage room (110) and takes out the hand with an empty hand. The refrigerator (100) can estimate the location information of the object (112) in the storage room (110) by using the movement of the user's (114) hand and arm, gaze, movement path, change in size of the object (112), or motion vector, etc., recognized from the first video.

[0133] According to one embodiment of the present disclosure, a refrigerator (100) can recognize from an input image that an object (112) is being released from a storage room (110). The refrigerator (100) can determine that the object (112) has been released if it recognizes from an input image that a user (114) is putting their hand into the storage room (110) with an empty hand, and recognizes that the user (114) is taking the object (112) out of the storage room (110).

[0134] Additionally, according to one embodiment of the present disclosure, the refrigerator (100) can recognize from an input image that an object (112) has been received and released from the storage room (110). The refrigerator (100) can determine that the first object has been received and the second object has been released when it recognizes that the user (114) has received the first object by holding it in their hand and placed it in the storage room (110), and then received the second object by holding it in the storage room (110) and taking their hand out of the storage room (110).

[0135] The refrigerator (100) can generate and store storage object information that stores keywords and location information related to objects stored in the storage room (100). When the refrigerator (100) recognizes the entry or exit of an object (112), it can update information about the entered or exited object (112) in the storage object information.

[0136] FIG. 10 is a drawing showing a plurality of storage compartment areas of a storage compartment of a refrigerator according to one embodiment of the present disclosure.

[0137] According to one embodiment of the present disclosure, the storage room (110) of the refrigerator (100) may include a plurality of storage room areas (1010a, 1010b, 1010c, 1010c, 1010d, 1010e, 1010f, 1010g, 1010h, 1010i, and 1010j). In the present disclosure, the plurality of storage room areas (1010a, 1010b, 1010c, 1010c, 1010d, 1010e, 1010f, 1010g, 1010h, 1010i, and 1010j) are collectively referred to as identification number 1010.

[0138] The storage room (110) may be divided into multiple storage room areas (1010) by shelves, drawers, partitions, or door baskets, etc. The multiple storage room areas (1010) may be defined by shelves, drawers, etc. Additionally, the multiple storage room areas (1010) may be divided into left or right sides within the same shelf. For example, the multiple storage room areas (1010) may be defined as the first shelf on the left (1010a), the second shelf on the left (1010c), the first shelf on the right (1010b), the second shelf on the right (1010d), the left vegetable room (1010i), or the right vegetable room (1010j).

[0139] According to one embodiment of the present disclosure, a refrigerator (100) can identify location information by identifying a plurality of storage room areas (1010) in which an object (112) is placed. Additionally, according to one embodiment of the present disclosure, a refrigerator (100) can identify location information by identifying the location within the storage room area (1010) in which an object (112) is placed. The location within the storage room area (1010) of the storage room (110) can be identified using three-dimensional coordinate information within the storage room area (1010).

[0140] FIG. 11 is a diagram illustrating a process of identifying location information of an object entering a storage room according to one embodiment of the present disclosure.

[0141] According to one embodiment of the present disclosure, the refrigerator (100) detects the gaze direction (1120) of the user (114) from the first input image (1110). The user (114) holds the object (112) in their hand and looks toward the direction in which the object (112) is to be placed. Accordingly, the refrigerator (100) can estimate location information for where the user (114) will place the object (112) within the storage room (110) based on the gaze direction (1120) of the user (114). For example, if the gaze direction (1120) of the user (114) is directed toward the upper right, the refrigerator (100) can estimate that the object (112) will be placed in at least one storage room area (1010) on the upper right side. For example, when the user (114)’s line of sight (1120) is facing the upper right side, the refrigerator (100) determines the second storage area (1010b) and the fourth storage area (1010c) on the upper right side as candidate storage areas, and can estimate that an object (112) will be placed in one of the candidate storage areas.

[0142] A refrigerator (100) can identify a gaze direction (1120) from a first input image (1110) using a gaze recognition algorithm. According to one embodiment of the present disclosure, the refrigerator (100) can identify a gaze direction (1120) by recognizing the face, eyes, and pupils of a user (1140) from the first input image (1110) and recognizing the direction in which the face is facing and the direction of the pupils.

[0143] Additionally, according to one embodiment of the present disclosure, the refrigerator (100) can recognize, from a second input image (1112) captured after the first input image (1110) was captured, the user (114) leans their body toward the storage room (110) of the refrigerator (100) and places an object (112) inside the storage room (110) by inserting their arm into the storage room (110). The refrigerator (100) can recognize at least one of the movement of the user's (114) arm, the motion of the user's (114) hand and arm, the angle at which the arm is inserted into the storage room (110), or the depth to which the arm is inserted into the storage room (110). The refrigerator (100) can estimate the location of the storage area (1010) where the object (112) is placed and the location within the storage area (1010) by using at least one of the movement of the user's (114) hand and arm, the motion of the hand and arm, or the angle at which the arm is inserted into the storage room (110). Additionally, the refrigerator (100) can estimate the depth of the object (112) within the storage area (1010) where the object (112) is placed based on the depth to which the user's (114) arm has entered the storage room (110).

[0144] FIG. 12 is a diagram illustrating the process of estimating location information where an object is received in a storage room according to one embodiment of the present disclosure.

[0145] According to one embodiment of the present disclosure, a refrigerator (100) recognizes the size (1220a, 1220b) of an object (112) from input images (1210a, 1210b) and can estimate the height at which the object (112) is placed in the storage room (110) based on the size (1220a, 1220b) of the object (112). The refrigerator (100) can recognize the object (112) from the input images (1210a, 1210b). The refrigerator (100) measures the size of the object (112) recognized from the input images (1210a, 1210b). According to one embodiment of the present disclosure, the size of the object (112) can be measured as the length (1220a, 122b) of one axis of the object (112). For example, the explanation is given by referring to the third input image (1210a), which is the input image at the first time point, and the fourth input image (1210b), which is the input image at the second time point after a predetermined amount of time has elapsed from the first time point. The length of one axis of the object (112) recognized from the third input image (1210a) is the first length (1220a). The length of one axis of the object (112) recognized from the fourth input image (1210b) is the second length (1220b). If the second length (1220b) is shorter than the first length (1220a), the refrigerator (100) determines that the object (112) has moved away from the camera (120), that is, in a downward direction. If the second length (1220b) is longer than the first length (1220a), the refrigerator (100) determines that the object (112) has moved in a direction closer to the camera (120), that is, in an upward direction. In this way, the refrigerator (100) can recognize the direction of movement of the object (112) in the up and down direction based on the change in size of the object (112) in the input images (1210a, 1210b).

[0146] FIG. 13 is a diagram showing the rotation of a 3D internal image in 3D space in a 3D vision GUI according to one embodiment of the present disclosure.

[0147] According to one embodiment of the present disclosure, a 3D vision GUI may include 3D internal images (1330a, 1330b) on which 3D object images are placed. FIG. 13 shows a first GUI view (1310a) and a second GUI view (1310b) of a 3D vision GUI. The first GUI view (1310a) displays a front view of the 3D internal image (1330a). When a first user input corresponding to a touch-and-drag gesture (1320a) is input in the first GUI view (1310a), the 3D vision GUI may change the display area and display direction by rotating the 3D internal image (1330a) in a virtual 3D space to correspond to the drag direction of the touch-and-drag gesture (1320a). The 3D vision GUI can display a second GUI view (1310b) in which the 3D interior image is rotated to correspond to a touch and drag gesture (1320a). The second GUI view (1310b) can display a 3D interior image corresponding to the right diagonal view of the 3D interior image. In the diagonal view, the space other than the space corresponding to the storage room (110) in the 3D space can be displayed by processing it in solid color or pattern. The refrigerator (100) can switch the display direction of the 3D object image and the 3D interior image in the second GUI view (1310b) in the 3D space. To do this, the refrigerator (100) can display the surface visible from the direction corresponding to the display direction in the 3D object image and the 3D interior image in the second GUI view (1310b). By changing the display direction, the 3D object image and 3D internal image can be displayed so that the side not visible in the first GUI view (1310a) is visible in the second GUI view (1310b).

[0148] If, while the second GUI view (1310b) is displayed, a first user input corresponding to an additional touch and drag gesture (1320b) is received, the refrigerator (100) can change the display area and display direction by rotating the 3D interior image in the direction corresponding to the additional touch and drag gesture (1320b).

[0149] FIG. 14 is a diagram illustrating the process of zooming in on a display area in a 3D vision GUI according to one embodiment of the present disclosure.

[0150] According to one embodiment of the present disclosure, a refrigerator (100) can perform a zoom-in operation to enlarge the display area of ​​a 3D interior image displayed in a 3D vision GUI based on a first user input in a 3D vision GUI. According to one embodiment of the present disclosure, when a first user input corresponding to a zoom-in gesture (1430) is received, the refrigerator (100) can enlarge and display a storage area (1010) corresponding to the location where the zoom-in gesture (1430) was entered.

[0151] Referring to FIG. 14, the refrigerator (100) can detect that a touch input corresponding to a zoom-in gesture (1430) is input through a touchscreen corresponding to a display (130) in step S1420. The zoom-in gesture (1430) is a touch input in which two fingers touch the touchscreen and drag to move the two finger touch points apart. When the zoom-in gesture (1430) is input, the refrigerator (100) can identify a storage area (1010) corresponding to the area where the zoom-in gesture (1430) was input. For example, as shown in the example illustrated in FIG. 14, the refrigerator (100) can identify that the zoom-in gesture (1430) was detected in a fourth storage area (1010d) while the entire area of ​​the storage room (110) is displayed in a third GUI view (1410a). The refrigerator (100) can identify the storage area (1010) where the zoom-in gesture (1430) is detected as the selected area.

[0152] The refrigerator (100) can enlarge and display the selected area where the zoom-in gesture (1430) is detected in step S1422. For example, the refrigerator (100) can display a 3D interior image that enlarges and displays the fourth storage area (1010d) in the fourth GUI view (1410b).

[0153] According to one embodiment of the present disclosure, when a touch-and-drag gesture (1440) input is detected in step S1424 while the fourth GUI view (1410b) is displayed, the refrigerator (100) can rotate the 3D interior image in a virtual 3D space to correspond to the drag direction of the touch-and-drag gesture (1440). The refrigerator (100) can rotate the 3D interior image according to the touch-and-drag gesture (1440) while the selected area is zoomed in. According to one embodiment of the present disclosure, the refrigerator (100) can rotate the 3D interior image without displaying areas other than the selected area.

[0154] FIG. 15 is a diagram illustrating a process of changing the display area or display direction of a 3D internal image in response to a first user input according to one embodiment of the present disclosure.

[0155] According to one embodiment of the present disclosure, when a storage area is selected by a first user input in a 3D vision GUI, the refrigerator (100) determines a display area and a display direction according to the first user input and displays a 3D interior image in the determined display area and display direction.

[0156] Referring to FIG. 15, in step S1502, the refrigerator (100) can select a storage area by a first user input. According to one embodiment of the present disclosure, the first user input may correspond to a touch input that selects one storage area (1010) in a 3D vision GUI. Additionally, according to one embodiment of the present disclosure, the first user input may be a user voice input received through a microphone in a 3D vision GUI.

[0157] In step S1504, the refrigerator (100) can change the display area to a selected storage area based on the first user input and determine the display direction of the selected storage area.

[0158] According to one embodiment of the present disclosure, the first user input may specify both the display area and the display direction. In this case, the refrigerator (100) determines the display area and the display direction according to the first user input. For example, when the 3D Vision GUI displays a front view of the storage room (110), if the first user input selects the second storage room area (1010b), the refrigerator (100) displays the second storage room area (1010b) in a zoomed-in state from the front view (1510a). Additionally, when the second storage room area (1010b) is in a zoomed-in state from the front view (1510a), if the first user input is received to change the second storage room area (1010b) to a side view (1510b), the refrigerator (100) changes the 3D interior image in the 3D Vision GUI to a side view (1510b) and displays it.

[0159] According to one embodiment of the present disclosure, the first user input is a user voice, and can specify both the display area and the display direction. If the user voice requests, "Show the first shelf on the right as a top view," the refrigerator (100) can determine the display area as the first shelf on the right and the display direction as a top view (1510c).

[0160] According to one embodiment of the present disclosure, the first user input may select only the display area and not select the display direction. In this case, the refrigerator (100) may determine the display area based on the first user input and automatically determine the display direction. According to one embodiment of the present disclosure, the refrigerator (100) may accumulate data regarding the display area and display direction selected by the previous user input and automatically determine the display direction based on the accumulated data. For example, the refrigerator (100) may learn the display direction selected with the highest frequency for each storage compartment area (1010). Based on the learned display direction data, the refrigerator (100) may select the display direction selected with the highest frequency for the display area selected by the first user input.

[0161] Once the display area and display direction are determined, in step S1506, the refrigerator (100) can display a 3D interior image in a 3D vision GUI with the determined display area and display direction.

[0162] FIG. 16 is a drawing showing the distance between a plurality of objects in a 3D vision GUI increased and displayed according to one embodiment of the present disclosure.

[0163] According to one embodiment of the present disclosure, the refrigerator (100) can be modified so that hidden objects (112) become visible by increasing the distance between objects (112) while the selected storage area is enlarged and displayed in a zoomed-in state. When additional first user input is received in the zoomed-in state, the refrigerator (100) can be modified so that hidden objects (112) become visible by increasing the distance between objects (112).

[0164] The refrigerator (100) can display the selected storage area (1010) in a zoomed-in state by enlarging it in the sixth GUI view (1610a). When a zoom-in gesture (1430) is input in the sixth GUI view (1610a), the refrigerator (100) can change and display the 3D interior image in the seventh GUI view (1610b) so that the hidden object (112) becomes visible by increasing the distance between the objects (112). According to one embodiment of the present disclosure, the refrigerator (100) can move the 3D object image of each object (112) to increase the distance between the objects (112) from a predetermined reference point (1620) within the displayed storage area (1010). For example, the refrigerator (100) can adjust the position of the 3D object image so that the overlapping or obscured object (112) is visible by increasing the distance from the reference point (1620) for the object (112) that is obscured or overlapping by another object (112) in the current display direction. The 3D object image of the obscured object (112) can be moved in a direction away from the reference point (1620).

[0165] According to one embodiment of the present disclosure, the reference point (1620) may correspond to the center of the lower surface of the storage room area (1010). Additionally, according to one embodiment of the present disclosure, the reference point (1620) may correspond to a preset point of the storage room area (1010).

[0166] According to one embodiment of the present disclosure, when a zoom-out gesture is input to reduce the distance between two fingers while the distance between objects (112) is displayed as increased, the refrigerator (100) can reduce the distance between objects (112). Additionally, according to one embodiment of the present disclosure, when a zoom-out gesture is input, the refrigerator (100) can display a 3D interior image in a sixth GUI view in the 3D vision GUI without adjusting the distance between objects (112).

[0167] The drawing in FIG. 16 shows the distance between objects (112) being increased while displaying a 3D internal image in a diagonal view in a 3D vision GUI.

[0168] FIG. 17 is a drawing showing the distance between a plurality of objects in a 3D vision GUI increased and displayed according to one embodiment of the present disclosure.

[0169] FIG. 17 is a drawing showing the distance between objects (112) being increased in a state (1710a) in which a selected storage area (1010) is displayed in a front view according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, when a zoom-in gesture (1430) is input in a state (1710a) in which a selected storage area (1010) is displayed in a front view, the refrigerator (100) can move the 3D object image for each object (112) so that the distance between objects (112) increases, as shown in 1710b. The refrigerator (100) can change or adjust the position of the 3D object image so that the object (112) that is obscured or overlapped by another object (112) in the front view is visible, by increasing the distance from the reference point (1620) of the obscured or overlapped object (112).

[0170] According to one embodiment of the present disclosure, when a touch-and-drag gesture (1320b) is input in a state (1710b) in which the distance between objects (112) is displayed to be far apart, the refrigerator (100) can change the display direction of the 3D interior image according to the drag direction of the touch-and-drag gesture (1320b). At this time, the refrigerator (100) can change the display direction of the 3D interior image while maintaining the state in which the distance between objects (112) is far apart.

[0171] FIG. 18 is a drawing showing the distance between a plurality of objects in a 3D vision GUI increased and displayed according to one embodiment of the present disclosure.

[0172] FIG. 18 is a drawing showing the distance between objects (112) being increased in a state (1810a) in which a selected storage area (1010) is displayed in a top view according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, when a zoom-in gesture (1430) is input in a state (1810a) in which a selected storage area (1010) is displayed in a top view, the refrigerator (100) can move the 3D object image for each object (112) so that the distance between objects (112) increases, as shown in 1810b. The refrigerator (100) can change or adjust the position of the 3D object image so that the object (112) that is obscured or overlapped by another object (112) in the top view is visible, by increasing the distance from the reference point (1620) of the obscured or overlapped object (112).

[0173] FIG. 19 is a diagram illustrating the process of determining the display area and display direction of a 3D internal image according to one embodiment of the present disclosure.

[0174] According to one embodiment of the present disclosure, a refrigerator (100) can determine a display area and a display direction of a 3D interior image in response to a first user input using a first artificial intelligence model (1910). If the first user input requests a 3D interior image without specifying a display direction, the refrigerator (100) can determine a display area and a display direction from the first artificial intelligence model (1910) based on the first user input and user identification information.

[0175] According to one embodiment of the present disclosure, the first artificial intelligence model (1910) takes the first user input and user identification information as input and outputs the display area and display direction of the 3D interior image. The first artificial intelligence model (1910) may be a model trained by a plurality of training data that takes the first user input and user identification information as input and outputs the display area and display direction of the 3D interior image. The refrigerator (100) collects training data during operation, and when more than a reference number of training data are collected, the first artificial intelligence model (1910) can be trained. Additionally, the refrigerator (100) can further collect training data during operation to reinforce the first artificial intelligence model (1910).

[0176] According to one embodiment of the present disclosure, the first artificial intelligence model (1910) may operate on a server. The refrigerator (100) transmits collected training data to the server, and the server may train the first artificial intelligence model (1910) using the collected training data. Additionally, if additional training data is collected, the server may perform reinforcement training on the first artificial intelligence model (1910). When the first artificial intelligence model (1910) operates on a server, the refrigerator (100) transmits the first user input and user identification information to the server, and the server may obtain the display area and display direction from the selected storage area information and user identification information using the first artificial intelligence model (1910). The server may transmit the display area and display direction to the refrigerator (100).

[0177] According to one embodiment of the present disclosure, the first user input may be an input requesting to show the storage room (110) without specifying a display area and a display direction. For example, when a user requests the refrigerator (100) using a voice recognition function, saying "Show me the inside of the refrigerator," the refrigerator (100) inputs the voice-recognized first user input to the first artificial intelligence model (1910) and obtains the display area and display direction output from the first artificial intelligence model (1910).

[0178] According to one embodiment of the present disclosure, the first user input may be an input that specifies only the display area without specifying the display direction. For example, if a user requests the refrigerator (100) using a voice recognition function, saying "Show me the second compartment from the left," the refrigerator (100) may recognize the storage compartment area selected by the user as the first user input. In this case, the first artificial intelligence model (1910) may determine the display area according to the first user input, and the display direction may be determined by the operation of the first artificial intelligence model (1910) based on the display area selected by the user and user identification information.

[0179] According to one embodiment of the present disclosure, a refrigerator (100) can preprocess a first user input and input it into a first artificial intelligence model (1910). For example, through a preprocessing process, the refrigerator (100) can recognize display area information or display direction information from the first user input and input the recognized display area information or display direction information into the first artificial intelligence model (1910) as the first user input.

[0180] The refrigerator (100) can obtain user identification information. According to one embodiment of the present disclosure, the refrigerator (100) can obtain user identification information using user voice or camera input image, etc.

[0181] According to one embodiment of the present disclosure, when a refrigerator (100) receives a first user input using a voice recognition function, it can obtain user identification information using the user's voice. The refrigerator (100) can collect voice pattern information for each user using the signal pattern of the user input received using the voice recognition function. The refrigerator (100) can obtain user identification information from the user's voice by comparing the collected voice pattern information with the user's voice received through the voice recognition function. When a user's voice with a new pattern that does not match the collected voice pattern information is recognized, the refrigerator (100) can generate new user identification information and collect and store voice pattern information.

[0182] According to one embodiment of the present disclosure, a refrigerator (100) can recognize a user's face from an input image captured by a camera (120) and obtain user identification information.

[0183] FIG. 20 is a block diagram showing the structure of a refrigerator according to one embodiment of the present disclosure.

[0184] According to one embodiment of the present disclosure, the refrigerator (100) may include a storage compartment (110), a camera (120), a processor (210), an input interface (212), a memory (214), a microphone (2010), a touchscreen (2020), and a speaker (2030). The touchscreen (2020) may correspond to the display (130) of FIG. 2.

[0185] In FIG. 20, to avoid duplication of explanation, the explanation focuses on the differences from the embodiment of FIG. 2 described earlier.

[0186] The refrigerator (100) may include a touchscreen (2020). The touchscreen (2020) may be placed on at least one door (410). The touchscreen (2020) may display a 3D vision GUI. Additionally, the touchscreen (2020) may receive user input by detecting a user's touch gesture. Touch gestures may include gestures such as tap, double tap, press, drag, pinch, or rotate. According to one embodiment of the present disclosure, touch gestures may include at least one of a touch-and-drag gesture of touching and dragging a point, a zoom-in gesture which is a pinch gesture that increases the distance between two fingers, a zoom-out gesture which is a pinch gesture that decreases the distance between two fingers, or a rotate gesture which rotates fingers while touching with two fingers.

[0187] The input interface (212) may include a microphone (2010). The microphone (2010) detects an audio signal. The processor (210) receives the audio signal detected from the microphone (2010). The processor (210) processes the audio signal and can recognize a user's command from the audio signal using a voice recognition function. The processor (210) can recognize the voice-recognized user command as a first user input.

[0188] According to one embodiment of the present disclosure, when a user voice that triggers a voice recognition function is recognized, the processor (210) can recognize a user command using the voice recognition function from the microphone (2010). For example, when the word "Hi Bixby" is recognized from the user voice, the processor (210) can recognize the user command by voice-recognizing an audio signal input after "Hi Bixby".

[0189] The speaker (2030) can output information or warning sounds related to the operation of the refrigerator (100) as an audio signal. According to one embodiment of the present disclosure, the speaker (2030) can output a voice message requesting storage object information from the user. The processor (210) can generate a voice message requesting information from the user and output it through the speaker (2030).

[0190] FIG. 21 is a diagram illustrating a process of obtaining user identification information according to one embodiment of the present disclosure.

[0191] According to one embodiment of the present disclosure, a refrigerator (100) can recognize a user's voice from an audio signal obtained using a microphone (2010) and obtain user identification information.

[0192] Referring to FIG. 21, the refrigerator (100) can recognize a user's voice from an audio signal in step S2102. The refrigerator (100) can determine whether the detected audio signal is a user's voice by using a microphone (2010). The refrigerator (100) can determine whether the audio signal is a user's voice by using characteristic information of a human voice. The characteristic information of a human voice may include a frequency range, a signal distribution according to frequency, or a pattern of the audio signal.

[0193] Next, in step S2104, the refrigerator (100) can identify a first user input from the user voice included in the audio signal. The refrigerator (100) can identify the first user input using a speech recognition function. For example, the refrigerator (100) can perform speech recognition using various types of speech-to-text (STT) algorithms. The refrigerator (100) can identify the content of the first user input based on the speech recognition result.

[0194] Additionally, in step S2106, the refrigerator (100) can obtain user identification information based on the user's voice. The refrigerator (100) can collect voice pattern information for each user by using the signal pattern of the input user input using a voice recognition function. The refrigerator (100) can obtain user identification information from the user's voice by comparing the collected voice pattern information with the input user's voice through the voice recognition function. If a user's voice with a new pattern that does not match the collected voice pattern information is recognized, the refrigerator (100) can generate new user identification information and collect and store voice pattern information.

[0195] The order of steps S2104 and S2106 is not limited to that shown in FIG. 21. According to one embodiment of the present disclosure, it is possible for step S2106 to be performed first and S2104 to be performed next. Additionally, according to one embodiment of the present disclosure, it is possible for steps S2104 and S2106 to be performed in parallel.

[0196] FIG. 22 is a diagram illustrating a process for obtaining storage object information and an example of storage object information according to one embodiment of the present disclosure.

[0197] According to one embodiment of the present disclosure, a refrigerator (100) can obtain storage object information (2210) from an input image (2220) captured by a camera (120). The refrigerator (100) can recognize an object (112) being received from the input image (2220). The refrigerator (100) can recognize keywords related to the recognized object (112). The refrigerator (100) can recognize keywords related to food. For example, keywords may include a name, quantity, condition, cooking condition, or product information. Product information may include a brand, manufacturer, or expiration date, etc.

[0198] The refrigerator (100) can recognize an object (112) from an input image (2220) and identify the name of the object (112). The name can be defined as a common name. The name can be identified, for example, as eggplant, tomato, shrimp, strawberry jam, or chicken.

[0199] The refrigerator (100) can recognize the number of objects (112) from the input image (2220). As previously explained with reference to FIG. 7, the number of objects (112) can be recognized by separating and recognizing the objects (112).

[0200] Additionally, according to one embodiment of the present disclosure, the refrigerator (100) can recognize the state of an object (112) from an input image (2220). For example, the refrigerator (100) can recognize the state of the object (112) based on the shape, color, or pattern of the object (112) in the input image (2220). The state of the object (112) can be defined, for example, as an unripe tomato, a peeled chestnut, a chopped onion, or shredded cabbage.

[0201] Additionally, according to one embodiment of the present disclosure, the refrigerator (100) can recognize the cooking state of an object (112) from an input image (2220). For example, the refrigerator (100) can recognize the cooking state of an object (112) based on the shape, color, or pattern of the object (112) in the input image (2220). The cooking state may be defined, for example, as grilled shrimp, fried food, spicy braised short ribs, or soy sauce braised short ribs.

[0202] Additionally, according to one embodiment of the present disclosure, if the object (112) is a ready-made product, the stored object information (2210) may include product information. The refrigerator (100) may recognize that the object (112) is a ready-made product based on characters, containers, colors, or patterns, etc., recognized from the object (112) in the input image (220). If the object (112) is a ready-made product, the refrigerator (100) recognizes product information from the input image (2220). The refrigerator (100) may recognize product information based on the recognized characters, containers, colors, or patterns, etc. For example, the refrigerator (100) may store feature information of each ready-made product (e.g., characters, containers, colors, or patterns, etc.) in advance, and obtain product information using the information recognized in the input image (2220) and the feature information of the ready-made product stored in advance. Additionally, for example, the server stores characteristic information of each ready-made product (e.g., characters, containers, colors, or patterns, etc.) in advance, and the refrigerator (100) transmits the information recognized from the input image (2220) and the pre-stored characteristic information of the ready-made product to the server, and the server can identify whether the object (112) recognized from the input image (2220) is a ready-made product and identify product information. The server transmits the acquired product information to the refrigerator (100), and the refrigerator (100) can acquire the product information.

[0203] According to one embodiment of the present disclosure, the storage object information (2210) may be stored in memory (214). Additionally, according to one embodiment of the present disclosure, the storage object information (2210) may be stored in a server.

[0204] According to one embodiment of the present disclosure, the storage object information (2210) may include location information of the object (112). The refrigerator (100) can obtain location information of the object (112) that has been brought into the storage room (110), as previously described with reference to FIGS. 9 to 12.

[0205] FIG. 23 is a flowchart illustrating the process of outputting location information of an object using voice recognition according to one embodiment of the present disclosure.

[0206] FIG. 24 is a diagram illustrating the process of outputting location information of an object using voice recognition according to one embodiment of the present disclosure.

[0207] According to one embodiment of the present disclosure, the refrigerator (100) can output information about the location of an object (112) stored in a storage room (110) based on user input received using voice recognition.

[0208] Referring to FIG. 23, the refrigerator (100) recognizes a user's voice from an audio signal input through a microphone (2010) in step S2302. The refrigerator (100) can recognize the user's voice using a voice recognition function as described above.

[0209] Referring to FIG. 24, a user (2412) can request location information of an object (112) from a refrigerator (100) through voice utterance (2410). For example, the user (2412) can say to the refrigerator (100), "Where is the gochujang?" The refrigerator (100) detects the voice utterance (2410) through a microphone (2010) and can recognize the user's voice from the detected voice utterance (2410).

[0210] Referring again to FIG. 23, the refrigerator (100) recognizes keywords related to food from the user's voice in step S2304. Keywords related to food may correspond to the name, quantity, condition, cooking condition, or product information of the food. According to one embodiment of the present disclosure, the refrigerator (100) may store keywords related to food in advance and recognize keywords related to food using the stored keywords.

[0211] As illustrated in FIG. 24, when a user (2412) speaks "Where is the gochujang?", the refrigerator (100) recognizes the keyword "gochujang", which is related to food, from the user's voice. The refrigerator (100) recognizes the word from the recognized user's voice and determines whether the recognized word is a keyword related to food, thereby recognizing the keyword related to food.

[0212] Referring again to FIG. 23, the refrigerator (100) searches for a keyword corresponding to a keyword recognized in the storage object information in step S2306. The refrigerator (100) can search for a keyword from the name, quantity, state, cooking state, or product information of the storage object information. For example, the refrigerator (100) can search for gochujang in the storage object information.

[0213] In step S2308, if an object corresponding to a keyword is found from the storage object information, the refrigerator (100) can output location information of the found object.

[0214] According to one embodiment of the present disclosure, a refrigerator (100) can output a voice message (2420) through a speaker (2030). When a keyword is searched from the stored object information, the refrigerator (100) can obtain location information of an object corresponding to the searched keyword. The refrigerator (100) can convert the obtained location information into a voice message (2420) and output it through the speaker (2030). For example, the refrigerator (100) can output a voice message (2420) saying, "Gochujang is on the second shelf from the right." Additionally, for example, the refrigerator (100) can output a voice message (2420) saying, "Gochujang is on the second shelf from the right, behind the strawberry jam."

[0215] Additionally, according to one embodiment of the present disclosure, the refrigerator (100) can display location information of an object corresponding to a keyword through a display (130). According to one embodiment of the present disclosure, the refrigerator (100) zooms in and displays a storage area containing an object requested by the user in the eighth GUI view (2432) of the 3D vision GUI (2430). Additionally, the refrigerator (100) can display the storage area containing the searched object in a display direction preferred by the user. Additionally, according to one embodiment of the present disclosure, the refrigerator (100) can apply a predetermined transparency to the remaining objects excluding the searched object (2436) in the ninth GUI view (2434) of the 3D vision GUI (2430), and the searched object (2436) can be displayed opaquely. For example, the refrigerator (100) can apply 50% transparency to the remaining objects excluding the searched object (2436). Additionally, according to one embodiment of the present disclosure, the refrigerator (100) may display a text tag (e.g., gochujang) corresponding to the retrieved object (2436) together around the object (2436).

[0216] FIG. 25 is a flowchart illustrating a process for obtaining keywords for unidentifiable objects in an input image according to one embodiment of the present disclosure.

[0217] According to one embodiment of the present disclosure, the refrigerator (100) may be unable to identify an object (112) from an input image (2520). For example, the refrigerator (100) may be unable to identify the object (112) from the input image (2520) when the object (112) is contained in a container, wrapped in a bag, or wrapped in foil. In such cases, the refrigerator (100) may obtain keywords for the object (112) by using a voice recognition function.

[0218] Referring to FIG. 25, the refrigerator (100) may determine in step S2502 that it is impossible to recognize a keyword for an object (112) from an input image (2520). For example, the refrigerator (100) may determine that the object (112) is impossible to identify from the input image (2520) when the object (112) is contained in a container, wrapped in a bag, or wrapped in foil.

[0219] In this case, the refrigerator (100) may output a request message requesting information about an object at step S2504. According to one embodiment of the present disclosure, the refrigerator (100) may output a request message requesting information about an object (112) to a user through a speaker (2030). For example, the refrigerator (100) may output a request message saying, "Please tell me the name of the food inside the container."

[0220] Next, in step S2506, the refrigerator (100) can recognize user voice input through the microphone (2010) in response to a request message. The refrigerator (100) can recognize user voice input through the microphone (2010) for a predetermined time (e.g., 30 seconds) after the request message is output.

[0221] In step S2508, the refrigerator (100) can recognize keywords related to food from the recognized user voice. If keywords related to food are recognized from the user voice, the refrigerator (100) can store the keywords as storage object information in step S2510. The refrigerator (100) can use the input image (2520) to estimate location information of the recognized object (112) within the storage chamber (110) and store it along with the keywords.

[0222] FIG. 26 is a diagram illustrating a process of generating a food image from a keyword according to one embodiment of the present disclosure.

[0223] According to one embodiment of the present disclosure, the refrigerator (100) may display a food image or text tag related to the object (112) on or around the 3D object image when the object (112) is not visible.

[0224] According to one embodiment of the present disclosure, as illustrated in FIG. 26, a refrigerator (100) can generate food images from keywords using a second artificial intelligence model (2610). The second artificial intelligence model (2610) takes keywords related to food as input and food images as output. The second artificial intelligence model (2610) can be trained by a plurality of training data that take keywords related to food as input and food images as output. The second artificial intelligence model (2610) can correspond to an image generation model.

[0225] According to one embodiment of the present disclosure, the second artificial intelligence model (2610) may receive keywords including a name, quantity, state, cooking state, or product information. The second artificial intelligence model (2610) may generate a food image by reflecting the received keywords. For example, if the keywords include "shrimp," "10 pieces," and "grilled shrimp," the second artificial intelligence model (2610) may generate a food image corresponding to grilled shrimp. Additionally, the second artificial intelligence model (2610) may generate a food image containing 10 or more grilled shrimp.

[0226] According to one embodiment of the present disclosure, the second artificial intelligence model (2610) can be executed in the processor (210) of the refrigerator (100).

[0227] Additionally, according to one embodiment of the present disclosure, the second artificial intelligence model (2610) may operate on a server. When the second artificial intelligence model (2610) operates on a server, the refrigerator (100) transmits keywords for an object (112) to the server, and the server can obtain a food image using the second artificial intelligence model (2610). The server can transmit the food image to the refrigerator (100).

[0228] FIG. 27 is a drawing showing a food image displayed on an object when the object is not visible, according to one embodiment of the present disclosure.

[0229] According to one embodiment of the present disclosure, in a first GUI view (2710), the refrigerator (100) displays a food image (2730) and a text tag (2732) generated for a first object (2734) together with the first object (2734). According to one embodiment of the present disclosure, when the first GUI view (2710) displays a 3D interior image without the food image (2730) and the text tag (2732), and a touch input by a user touching the object (2734) is detected, the food image (2730) and the text tag (2732) can be displayed on the object (2734).

[0230] According to one embodiment of the present disclosure, a refrigerator (100) may display a food image (2730) and a text tag (2732) on an object whose interior is not visible in the first GUI view (2720). Additionally, the refrigerator (100) may display only a text tag (2732) without a food image (2730) on an object whose interior is visible in the first GUI view (2720).

[0231] FIG. 28 is a diagram illustrating the process of obtaining keywords for delivery food according to one embodiment of the present disclosure.

[0232] According to one embodiment of the present disclosure, a refrigerator (100) can obtain a keyword from a delivery application when an object (112) recognized from an input image corresponds to delivery food.

[0233] Referring to FIG. 28, the refrigerator (100) can recognize an object (112) from an input image in step S302.

[0234] In step S2802, the refrigerator (100) can determine whether the recognized object (112) is delivery food. The refrigerator (100) can determine whether the object (112) is delivery food based on the container of the object (112) recognized from the input image. The refrigerator (100) can recognize that the object (112) is delivery food by using text, a picture, a logo, a color, or a pattern, etc., on the container of the object (112).

[0235] If the refrigerator (100) determines that the object (112) is not delivery food, it can proceed to step S304.

[0236] If the refrigerator (100) determines that the object (112) is delivery food, in step S2804, it can obtain order information from a food delivery application. According to one embodiment of the present disclosure, the refrigerator (100) may have at least one food delivery application installed or be linked with at least one food delivery application. The refrigerator (100) can obtain order information from the installed or linked food delivery application.

[0237] Next, in step S2806, the refrigerator (100) can obtain keywords related to the object (112) from the obtained order information. Additionally, the refrigerator (100) can obtain at least one of a food image or a text tag from the obtained order information. For example, a food image of a delivery food may correspond to a food image included in the order information. Additionally, the refrigerator (100) can obtain information such as the order date and menu name from the order information and store it in the stored food information.

[0238] FIG. 29 is a drawing showing a refrigerator, an external device, and a server according to one embodiment of the present disclosure.

[0239] According to one embodiment of the present disclosure, a refrigerator (100) communicates with an external device (2910) and a server (2920) through a communication module (not shown). The refrigerator (100) may be connected to other home appliances, the external device (2910), or the server (2920) through a network (NET).

[0240] The server (2920) can manage user account information and information about the refrigerator (100) connected to the user account. For example, a user can create a user account by accessing the server (2920) through an external device (2910). The user account can be identified by an ID and password set by the user. The server (2920) can register the refrigerator (100) to the user account according to a set procedure. For example, the server (2920) can register the refrigerator (100) by linking the identification information of the refrigerator (100) (e.g., serial number or MAC address) to the user account.

[0241] The external device (2910) may include a communication module capable of communicating with the refrigerator (100) and the server (2920), a user interface that receives user input or outputs information to the user, at least one processor that controls the operation of the external device (2910), and at least one memory in which a program for controlling the operation of the external device (2910) is stored.

[0242] The external device (2910) may be carried by the user or placed in the user's home or office, etc. The external device (2910) may include, for example, a personal computer, a terminal, a portable telephone, a smartphone, a handheld device, a wearable device, etc., but is not limited thereto.

[0243] A program (e.g., an application) for controlling the refrigerator (100) may be stored in the memory of the external device (2910). The external device (2910) may be sold with the application for controlling the refrigerator (100) installed, or it may be sold without the application installed. If the external device (2910) is sold without the application for controlling the refrigerator (100) installed, the user may download the application from an external server that provides the application and install it on the external device (2910).

[0244] The user can control the refrigerator (100) using an application installed on an external device (2910). For example, when the user runs an application installed on the external device (2910), identification information of the refrigerator (100) connected to the same user account as the external device (2910) may appear in the application execution window. The user can perform desired control on the refrigerator (100) through the application execution window. When the user inputs a control command for the refrigerator (100) through the application execution window, the external device (2910) may transmit the control command directly to the refrigerator (100) via a local area network, or it may transmit the control command to the refrigerator (100) via a server (2920).

[0245] The application of the external device (2910) can receive various user inputs for controlling the refrigerator (100). The application provides a Graphic User Interface (GUI) for receiving various user inputs and receives user inputs through the GUI. The external device (2910) communicates with the server (2920) and updates the status information of the refrigerator (100) and provides it through the application. Additionally, the external device (2910) communicates with the server (2920) and transmits the user input received through the application to the refrigerator (100).

[0246] Additionally, the application can identify the network of the Wi-Fi network to which the external device (2910) is connected and transmit the identification information of the Wi-Fi network to the refrigerator (100). The external device (2910) can transmit the identification information of the Wi-Fi network to the refrigerator (100) in response to a request from the refrigerator (100) via the server (2920).

[0247] A network (NET) may include both wired networks and wireless networks. Wired networks include cable networks or telephone networks, etc., and wireless networks may include all networks that transmit and receive signals via radio waves. Wired networks and wireless networks may be connected to each other.

[0248] A network (NET) may include a wide area network (WAN) such as the Internet, a local area network (LAN) formed around an access point (AP), and a wireless personal area network (WPAN) that does not pass through an access point. A wireless personal area network may include, but is not limited to, Bluetooth (IEEE 802.15.1), Zigbee (IEEE 802.15.4), Wi-Fi Direct, Near Field Communication (NFC), Z-Wave, etc.

[0249] The access point (AP) can connect the local network (LAN) to which the refrigerator (100) and external device (2910) are connected to the wide area network (WAN) to which the server (2920) is connected. The refrigerator (100) or external device (2910) can be connected to the server (2930) via the wide area network (WAN).

[0250] An AP may include a device that enables devices to be connected in a computer network using relevant standards utilizing Wi-Fi.

[0251] According to embodiments of the present disclosure, the AP may include a hardware-implemented AP and a software-implemented AP.

[0252] For example, an AP can relay data between wireless devices and wired devices on a network. However, it is not limited to this; an AP can also relay data between wired devices or between wireless devices. Meanwhile, an AP can also be referred to as a relay device.

[0253] The access point (AP) can communicate with the refrigerator (100) and external device (2910) using wireless communication such as Wi-Fi (Wi-Fi™, IEEE 802.11), and can connect to a wide area network (WAN) using wired communication.

[0254] The refrigerator (100) can transmit information regarding operation or status to the server (2920) via a network (NET). For example, the refrigerator (100) can transmit information regarding operation or status to the server (2920) via Wi-Fi (Wi-Fi™, IEEE 802.11) communication.

[0255] If the refrigerator (100) is not equipped with a Wi-Fi communication module, the refrigerator (100) can transmit information regarding operation or status to the server (2920) through another home appliance equipped with a Wi-Fi communication module. For example, if the refrigerator (100) transmits information regarding operation or status to another home appliance via a short-range wireless network (e.g., Bluetooth Low Energy (BLE) communication), the other home appliance can transmit information regarding the operation or status of the refrigerator (100) to the server (2920). Additionally, for example, if the refrigerator (100) is not equipped with a Wi-Fi communication module, the refrigerator (100) can be connected to a communication relay device via a wired connection and perform Wi-Fi communication through the communication relay device.

[0256] The refrigerator (100) may provide information regarding the operation or status of the refrigerator (100) to the server (2920) upon prior approval by the user. The transmission of information to the server (2920) may occur when a request is received from the server (2920), when a specific event occurs in the refrigerator (100), or periodically or in real time.

[0257] When the server (2920) receives information regarding operation or status from the refrigerator (100), it can update previously stored information regarding the refrigerator (100). The server (2920) can transmit information regarding operation or status of the refrigerator (100) to an external device (2910) via a network (NET).

[0258] The server (2920) can transmit information regarding the operation or status of the refrigerator (100) to the external device (2910) when a request is received from the external device (2910). For example, when a user runs an application connected to the server (2920) on the external device (2910), the external device (2910) can request and receive information regarding the operation or status of the refrigerator (100) from the server (2920) through the application. The server (2920) may also transmit information regarding the operation or status of the refrigerator (100) to the external device (2910) in real time when information regarding the operation or status of the refrigerator (100) is received from the refrigerator (100). The server (2920) may also periodically transmit information regarding the operation or status of the refrigerator (100) to the external device (2910). The external device (2910) may transmit information regarding the operation or status of the refrigerator (100) to the user by displaying information regarding the operation or status of the refrigerator (100) in the application execution window.

[0259] The refrigerator (100) can obtain various information from the server (2920) and provide the obtained information to the user. Additionally, the refrigerator (100) can receive a file from the server (2920) for updating the installed software or data related to the installed software, and can update the installed software or data related to the installed software based on the received file.

[0260] The refrigerator (100) can operate according to control commands received from the server (2920). For example, if the refrigerator (100) has obtained prior approval from a user to operate according to control commands from the server (2920) even without user input, the refrigerator (100) can operate according to control commands received from the server (2920). The control commands received from the server (2920) may include, but are not limited to, control commands entered by the user through an external device (2910) or control commands generated by the server (2920) based on pre-set conditions.

[0261] According to one embodiment of the present disclosure, a 3D vision GUI may be provided through an application of an external device (2910).

[0262] Additionally, according to one embodiment of the present disclosure, information or keywords related to an object may be input through an application of an external device (2910). The information or keywords related to an object input through an application of an external device (2910) may be transmitted to a refrigerator (100) through a server (2920).

[0263] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0264] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0265] According to one aspect of an embodiment of the present disclosure, a refrigerator is provided. The refrigerator includes a storage compartment for receiving an object that has been received, a camera that captures at least a portion of the storage compartment, an input interface that receives user input, a display, at least one processor, and a memory that stores a plurality of instructions. When the plurality of instructions are executed selectively or collectively by at least one processor, the refrigerator recognizes an object that has been received into the storage compartment from an input image captured by the camera, converts a 2D (2-Dimension) object image of the recognized object obtained from the input image into a 3D (3-Dimension) object image, places the converted 3D object image on a 3D storage image, and when a first user input is received to manipulate the 3D storage image while the 3D storage image is displayed on the display, changes at least one of the display area or display direction of the 3D storage image.

[0266] According to one embodiment of the present disclosure, a camera may be positioned to photograph the front of the opening of the storage room.

[0267] In addition, according to one embodiment of the present disclosure, an incoming object enters the storage room while being held in the user's hand, and the refrigerator can estimate the final position information placed within the storage room based on at least one of the user's arm angle, arm motion, user's gaze, the object's motion vector, or the user's arm motion vector from an input image.

[0268] Additionally, according to one embodiment of the present disclosure, the storage room includes a plurality of storage areas defined by at least one of a shelf, a partition, or a drawer, and the refrigerator can identify the storage area in which an object is stored among the plurality of storage areas.

[0269] Additionally, according to one embodiment of the present disclosure, the refrigerator can identify the depth at which an incoming object is placed within the storage compartment based on the depth at which a user's arm has entered the storage compartment.

[0270] Additionally, according to one embodiment of the present disclosure, the display and input interface is a touchscreen, and the refrigerator displays a 3D interior image in which a converted 3D object image is placed on the touchscreen, and when a first user input corresponding to a touch input that drags an area where the 3D interior image is displayed is received, the display direction of the 3D interior image can be rotated in 3D space to correspond to the drag direction of the touch input.

[0271] Additionally, according to one embodiment of the present disclosure, the refrigerator generates learned display direction data by learning the frequency of the selected display direction based on the history of the display area and display direction selected by the user, and when a first user input is received selecting a part of the storage room on the displayed 3D interior image, the refrigerator determines the display area and display direction of the 3D interior image to be displayed in response to the first user input based on the learned display direction data, and can display the 3D interior image on the display in the determined display area and display direction.

[0272] In addition, according to one embodiment of the present disclosure, a refrigerator acquires user identification information that has entered a first user input, inputs the first user input and user identification information into a first artificial intelligence model, and determines the display area and display direction of a 3D interior image to be displayed in response to the first user input using the display area and display direction output from the first artificial intelligence model, and the first artificial intelligence model can be trained using training data that takes the first user input and user identification information as input and the display area and display direction as output.

[0273] Additionally, according to one embodiment of the present disclosure, the input interface further includes a microphone for detecting an audio signal, and the refrigerator can recognize a user's voice from the detected audio signal, identify a first user input, and obtain user identification information based on the user's voice.

[0274] In addition, according to one embodiment of the present disclosure, the refrigerator can acquire at least one keyword related to an object based on the input image for an object recognized from an input image, and store storage object information including at least one keyword and location information for each object.

[0275] Additionally, according to one embodiment of the present disclosure, the input interface further includes a microphone for detecting an audio signal, and the refrigerator can recognize a user voice from the detected audio signal, recognize keywords related to food from the recognized user voice, search for an object corresponding to the recognized keywords based on storage object information, and output location information of the searched object.

[0276] Additionally, according to one embodiment of the present disclosure, the refrigerator further includes a speaker, and when the refrigerator cannot recognize a keyword of an incoming object from an input image, it outputs a request message requesting information of an incoming object through the speaker, and in response to the request message, recognizes food input by a user through a microphone, recognizes at least one keyword related to food from the recognized user voice, and can store the recognized at least one keyword as storage object information.

[0277] In addition, according to one embodiment of the present disclosure, the refrigerator can generate a food image corresponding to at least one recognized keyword using a second artificial intelligence model and display the generated food image on a 3D object image.

[0278] Additionally, according to one embodiment of the present disclosure, keywords related to food may include at least one of the name, quantity, condition, cooking condition, brand, manufacturer, or expiration date of the food.

[0279] In addition, according to one embodiment of the present disclosure, the refrigerator determines whether a recognized object is delivery food, and if the recognized object is delivery food, obtains order information corresponding to the recognized object from a food delivery application, and can store at least one of the food image, order date, or menu name of the delivery food included in the order information as at least one keyword.

[0280] In addition, according to one embodiment of the present disclosure, the refrigerator can display a food image of a delivered food on a 3D object image.

[0281] Additionally, according to one embodiment of the present disclosure, the storage room includes a plurality of storage areas defined by at least one of a shelf, a partition, or a drawer, and the refrigerator can display a 3D interior image on a display to enlarge the selected storage area and exclude the unselected storage area from the display area, and change the arrangement of the plurality of objects on the display so that hidden objects are visible by increasing the distance between the plurality of objects within the selected storage area.

[0282] Additionally, according to one aspect of an embodiment of the present disclosure, a refrigerator control method is provided. The refrigerator control method includes the steps of: recognizing an object that has entered the storage room from an input image captured by a camera placed in the storage room of the refrigerator; converting a 2D object image of the recognized object obtained from the input image into a 3D object image; placing the converted 3D object image on a 3D interior image; and, when a first user input that manipulates the 3D interior image is received while the 3D interior image is displayed, changing at least one of the display area or display direction of the 3D interior image.

[0283] According to one embodiment of the present disclosure, a camera may be positioned to photograph the front of the opening of the storage room.

[0284] In addition, according to one aspect of one embodiment of the present disclosure, a computer-readable recording medium is provided on which a program for performing a refrigerator control method on a computer is recorded.

Claims

1. In the refrigerator (100), Storage room (110) for accommodating incoming objects; A camera (120) that photographs at least a portion of the storage room (110); Input interface (212) for receiving user input; Display (130); At least one processor (210); and The refrigerator (100) includes a memory (214) for storing a plurality of instructions, and when the plurality of instructions are executed selectively or collectively by the at least one processor (210), the refrigerator (100), From the input image captured by the camera (120), an object entered into the storage room (110) is recognized, and The 2D (2-Dimension) object image of the recognized object obtained from the input image is converted into a 3D (3-Dimension) object image, and The above converted 3D object image is placed on the 3D interior image, and A refrigerator (100) that changes at least one of the display area or display direction of the 3D interior image when a first user input for manipulating the 3D interior image is received while the 3D interior image is displayed on the display (130).

2. In Paragraph 1, The above camera (120) is positioned to photograph the front of the opening of the above storage room (110), in a refrigerator (100).

3. In Paragraph 1 or 2, The above-mentioned received object enters the storage room (110) while being held in the user's hand, and The above refrigerator (100) is, A refrigerator (100) that estimates final position information within the storage room based on at least one of the angle of the user's arm holding the received object, arm motion, user's gaze, motion vector of the object, or motion vector of the user's arm from the input image.

4. In Paragraph 3, The above storage room (110) includes a plurality of storage areas defined by at least one of a shelf, a partition, or a drawer, and The above refrigerator (100) is, A refrigerator (100) that identifies the storage area in which the received object is received among the plurality of storage areas.

5. In Paragraph 3, The above refrigerator (100) is, A refrigerator (100) that identifies the depth at which the incoming object is placed within the storage room (110) based on the depth at which the user’s arm has entered the storage room.

6. In any one of paragraphs 1 through 5, The above display (130) and the above input interface (212) are touchscreens (2020), and The above refrigerator (100) is, Displays the 3D interior image on which the converted 3D object image is placed on the touchscreen (2020), and A refrigerator (100) that, when the first user input corresponding to a touch input that drags an area where the above 3D interior image is displayed is received, rotates the display direction of the above 3D interior image in 3D space to correspond to the drag direction of the touch input.

7. In any one of paragraphs 1 through 6, The above refrigerator (100) is, Based on the history of the display area and display direction selected by the user, the frequency of the selected display direction is learned to generate learned display direction data, and When the first user input selecting a portion of the storage room on the above-mentioned 3D interior image is received, the display area of ​​the 3D interior image to be displayed in response to the first user input and the display direction are determined based on the learned display direction data, and A refrigerator (100) that displays the 3D interior image on the display in the determined display area and display direction.

8. In Paragraph 7, The above refrigerator (100) is, Obtain user identification information that has entered the above first user input, and The first user input and the user identification information are input into the first artificial intelligence model (1910), and the display area and the display direction output from the first artificial intelligence model (1910) are used to determine the display area and the display direction of the 3D internal image to be displayed in response to the first user input. The above first artificial intelligence model (1910) is a refrigerator (100) that is trained using training data that takes the above first user input and the above user identification information as inputs and the above display area and the above display direction as outputs.

9. In any one of paragraphs 1 through 8, The above refrigerator (100) is, For an object recognized from the above input image, at least one keyword related to the object is obtained based on the above input image, and A refrigerator (100) that stores storage object information including at least one keyword and location information for each object.

10. In Paragraph 9, The above input interface (212) further includes a microphone (2010) for detecting an audio signal, and The above refrigerator (100) is, Recognizing the user voice from the detected audio signal, Recognizing keywords related to food from the above-mentioned recognized user voice, and Based on the above stored object information, search for an object corresponding to the above recognized keyword, and A refrigerator (100) that outputs location information of the above-mentioned searched object.

11. In Paragraph 10, The above refrigerator (100) further includes a speaker (2030), The above refrigerator (100) is, If the keyword of the received object cannot be recognized from the input image, a request message requesting information about the received object is output through the speaker, and In response to the above request message, the user's food input through the microphone (2010) is recognized, and Recognizing at least one keyword related to food from the above-mentioned recognized user voice, and A refrigerator (100) that stores at least one recognized keyword as the storage object information.

12. In Paragraph 9, The above refrigerator (100) is, Determining whether the above-mentioned recognized object is delivery food, and If the above-mentioned recognized object is delivery food, order information corresponding to the above-mentioned recognized object is obtained from the food delivery application, and A refrigerator (100) that stores at least one of the food image, order date, or menu name of the delivery food included in the above order information as at least one keyword.

13. In any one of paragraphs 1 through 12, The above storage room (110) includes a plurality of storage areas defined by at least one of a shelf, a partition, or a drawer, and The above refrigerator (100) is, When the first user input selecting one of the plurality of storage areas is received, The 3D internal image is displayed on the display (130) to enlarge the selected storage area and exclude the unselected storage area from the display area, and A refrigerator (100) that displays on the display (130) the arrangement of the plurality of objects so that the hidden object is visible by increasing the distance between the plurality of objects within the selected storage area.

14. In a method for controlling a refrigerator, A step of recognizing an object entered into the storage room from an input image captured by a camera placed in the storage room of the refrigerator; A step of converting a 2D object image of the recognized object obtained from the input image into a 3D object image; A step of placing the converted 3D object image onto a 3D internal image; and A refrigerator control method comprising the step of changing at least one of the display area or display direction of the 3D interior image when a first user input for manipulating the 3D interior image is received while the 3D interior image is displayed.

15. A computer-readable recording medium having a program recorded thereon for performing the method of paragraph 14 on a computer.

Images