Refrigerator that recommends storage location of food and control method therefor

Abstract

Disclosed is a refrigerator. The refrigerator according to the present disclosure may comprise a camera, a memory storing instructions, and at least one processor including processing circuitry, wherein, when individually or collectively executed by the at least one processor, the refrigerator may: acquire a first image through the camera, if a door of the refrigerator is opened; identify, on the basis of the first image, the type and size of an object put into the refrigerator; identify a first storage area corresponding to the object from among a plurality of storage areas of the refrigerator, on the basis of the type of food stored in each of the plurality of storage areas, the size of a remaining space, and the type and size of the object; and provide a first notification for guiding the first storage area.

Description

Refrigerator recommending food storage locations and its control method

[0001] The present disclosure relates to a refrigerator that recommends an optimal storage location and a method for controlling the same.

[0002] As refrigerators become larger and the types of food stored within them become more diverse, users may experience difficulties in organizing their food neatly. Additionally, users may struggle to manage a large variety and quantity of food stored in the refrigerator.

[0003] Accordingly, there was a need for a method to identify the types of food entering or leaving the refrigerator, organize and manage the types and locations of food stored in each storage area of ​​the refrigerator in a database, and recommend the optimal storage location when a user enters food.

[0004] A refrigerator is disclosed. The refrigerator according to the present disclosure comprises at least one processor including a camera, a memory for storing instructions, and a processing circuitry, and when executed individually or collectively by the at least one processor, when the door of the refrigerator is opened, the refrigerator acquires a first image through the camera, identifies the type and size of an object being placed in the refrigerator based on the first image, identifies a first storage area corresponding to the object among the plurality of storage areas based on the type of food stored in each of the plurality of storage areas of the refrigerator and the size of the remaining space, and the type and size of the object, and provides a first notification for guiding the first storage area.

[0005] The above camera can be movably installed on the refrigerator so as to protrude outward from the refrigerator.

[0006] Additionally, when the above instructions are executed individually or collectively by the at least one processor, the refrigerator identifies a second storage area in which the object is stored among the plurality of storage areas based on the first image, and identifies the protrusion length of the camera based on the second storage area,

[0007] Based on the above protrusion length, a second image is acquired through the camera, and based on the second image, the storage location of the object stored in the second storage area is identified. If the object is identified as being stored in the second storage area, information regarding the type of food and the size of the remaining space corresponding to the second storage area stored in the memory can be updated based on the storage location of the object.

[0008] Additionally, when the above instructions are executed individually or collectively by the at least one processor, the refrigerator may provide a second notification to guide the first storage area if the second storage area is identified as being different from the first storage area.

[0009] Additionally, when the above instructions are executed individually or collectively by the at least one processor, the refrigerator may identify a first protruding length corresponding to the third storage area as the protruding length of the camera when it is identified that the object is entered into the third storage area among the plurality of storage areas, and identify a second protruding length corresponding to the fourth storage area as the protruding length of the camera when it is identified that the object is entered into the fourth storage area among the plurality of storage areas. In this case, the height of the third storage area in the refrigerator may be lower than the height of the fourth storage area, and the first protruding length may be shorter than the second protruding length.

[0010] Additionally, when the above instructions are executed individually or collectively by the at least one processor, the refrigerator may detect a plurality of edges corresponding to a plurality of storage areas in an image acquired through the camera, and identify a storage area among the plurality of storage areas where an object is placed in the refrigerator based on the plurality of edges.

[0011] Additionally, when the above instructions are executed individually or collectively by the at least one processor, the refrigerator may identify the size of the remaining space of the dedicated storage area based on the size of the object stored in the dedicated storage area and the size of the dedicated storage area when it is identified that the refrigerator has a dedicated storage area for the object provided in the refrigerator, identify whether the object can be stored in the dedicated storage area based on the size of the object and the size of the remaining space of the dedicated storage area, and when it is identified that the object can be stored in the dedicated storage area, identify that the dedicated storage area is the first storage area and provide a notification to allow the user to store the object in the first storage area.

[0012] Additionally, if it is determined that the object cannot be stored in the dedicated storage area or that there is no dedicated storage area for the object, the storage area among the multiple storage areas in which an object of the same type as the object is stored is identified based on the type of the object stored in each of the multiple storage areas; the size of the object stored in the storage area in which an object of the same type as the object is stored is identified based on the size of the object stored in the storage area in which an object of the same type as the object is stored and the size of the storage area in which an object of the same type as the object is stored is identified based on the size of the object and the size of the remaining space in the storage area in which an object of the same type as the object is stored is identified based on the size of the object and the size of the remaining space in the storage area in which an object of the same type as the object is stored is identified based on the size of the object and the size of the remaining space in the storage area in which an object of the same type as the object is stored is identified based on the size of the object and the size of the remaining space in the storage area in which an object of the same type as the object is stored is identified as the first storage area, and a notification can be provided to allow the user to store the object in the first storage area.

[0013] Additionally, when the above instructions are executed individually or collectively by the at least one processor, if the refrigerator identifies that the object cannot be stored in a storage area where an object of the same type as the object is stored, or that there is no storage area where an object of the same type as the object is stored, it identifies a storage area among the plurality of storage areas where the object has a history of being stored based on the type of the object, identifies the size of the object stored in the storage area where the object has a history of being stored based on the size of the object stored in the storage area where the object has a history of being stored and the size of the remaining space in the storage area where the object has a history of being stored based on the size of the object and the size of the remaining space in the storage area where the object has a history of being stored based on the size of the object and the size of the remaining space in the storage area where the object has a history of being stored, and if it is identified that the object can be stored in the storage area where the object has a history of being stored, it identifies the storage area where the object has a history of being stored as a first storage area and provides a notification to the user to store the object in the first storage area.

[0014] Additionally, when the above instructions are executed individually or collectively by the at least one processor, the refrigerator identifies a first size of the object based on the first image, identifies a storage area among a plurality of storage areas of the refrigerator into which the object is placed, and identifies a second size of the object by applying a weight corresponding to the identified storage area based on information stored in the memory to the first size, and the greater the height of the storage area, the smaller the value of the weight applied to the first size may be.

[0015] Additionally, when the above instructions are executed individually or collectively by the at least one processor, if the refrigerator identifies that the size of the object is not identified based on the first image, it may identify the size of the object based on the similarity ratio between the object and the storage area in which the object is stored.

[0016] A method for controlling a refrigerator may include the steps of: acquiring a first image through a camera when the door of the refrigerator is opened; identifying the type and size of an object being placed in the refrigerator based on the first image; identifying a first storage area corresponding to an object among the plurality of storage areas based on the type of food stored in each of the plurality of storage areas of the refrigerator and the size of the remaining space, and the type and size of the object.

[0017] In addition, in a non-transient computer-readable recording medium that stores one or more instructions executed by a processor of the refrigerator to perform an operation of the refrigerator of the present disclosure, the operation may include the steps of: acquiring a first image through a camera when the door of the refrigerator is opened; identifying the type and size of an object being placed in the refrigerator based on the first image; identifying a first storage area corresponding to the object among the plurality of storage areas based on the type and size of the food stored in each of the plurality of storage areas of the refrigerator and the size of the remaining space of the object, and providing a first notification to guide the first storage area.

[0018] FIG. 1 is a drawing for explaining the operation of a refrigerator according to at least one embodiment of the present disclosure.

[0019] FIG. 2 is a drawing for explaining the exterior of a refrigerator according to at least one embodiment of the present disclosure.

[0020] FIG. 3 is a block diagram illustrating the configuration of a refrigerator according to at least one embodiment of the present disclosure.

[0021] FIG. 4 is a block diagram illustrating the detailed configuration of a refrigerator according to at least one embodiment of the present disclosure.

[0022] FIGS. 5A and FIGS. 5B are flowcharts for explaining the operation of a refrigerator according to at least one embodiment of the present disclosure.

[0023] FIG. 6 is a drawing for explaining a tank according to at least one embodiment of the present disclosure.

[0024] FIG. 7 is a drawing for explaining the shooting area of ​​a camera according to at least one embodiment of the present disclosure.

[0025] FIG. 8 is a drawing for explaining a method for identifying the storage location of an object being received according to at least one embodiment of the present disclosure.

[0026] FIG. 9 is a drawing for explaining the protrusion length and angle of a camera according to the position of an object according to at least one embodiment of the present disclosure.

[0027] FIGS. 10a and FIGS. 10b are drawings for illustrating a method for identifying the length of an object being received according to at least one embodiment of the present disclosure.

[0028] FIGS. 11a and 11b are drawings for illustrating a user interface that is updated as an object is received according to at least one embodiment of the present disclosure.

[0029] The various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments.

[0030] In relation to the description of the drawings, similar reference numerals may be used for similar or related components.

[0031] The singular form of the noun corresponding to the item may include one or multiple items, unless the relevant context clearly indicates otherwise.

[0032] In the present disclosure, each of the phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0033] The term “and / or” includes a combination of multiple related described components or any of the multiple related described components.

[0034] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0035] Additionally, terms such as 'front,' 'rear,' 'top,' 'bottom,' 'side,' 'left,' 'right,' 'top,' and 'bottom' used in this disclosure are defined based on the drawings, and the shape and location of each component are not limited by these terms.

[0036] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this disclosure, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0037] When it is said that a component is "connected," "combined," "supported," or "in contact" with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0038] When it is said that a component is located "on" another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0039] A refrigerator according to one embodiment may include a main body.

[0040] The "main body" may include an inner body, an outer body positioned on the outside of the inner body, and an insulating material provided between the inner body and the outer body.

[0041] The "inner body" may include at least one of a case, plate, panel, or liner forming a storage chamber. The inner body may be formed as a single body or may be formed by assembling multiple plates. The "outer body" may form the exterior of the main body and may be coupled to the outer side of the inner body so that an insulating material is disposed between the inner body and the outer body.

[0042] The "insulating material" can insulate the interior and exterior of the storage room so that the temperature inside the storage room is maintained at a set appropriate temperature without being affected by the external environment. According to one embodiment, the insulating material may include a foamed insulating material. The foamed insulating material can be formed by injecting and foaming urethane foam, which is a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0043] According to one embodiment, the insulation material may additionally include a vacuum insulation material in addition to a foam insulation material, or the insulation material may consist solely of a vacuum insulation material instead of a foam insulation material. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation material or vacuum insulation material described above and may include various materials that can be used for insulation.

[0044] The "storage room" may include a space defined by an internal structure. The storage room may further include an internal structure defining a space corresponding to the storage room. Various items such as food, medicine, and cosmetics may be stored in the storage room, and the storage room may be formed so that at least one side is open to allow for the retrieval and retrieval of items.

[0045] A refrigerator may include one or more storage compartments. When two or more storage compartments are formed in a refrigerator, each storage compartment may have a different use and may be maintained at a different temperature. To this end, each storage compartment may be partitioned from one another by a partition containing insulation.

[0046] The storage room may be provided to be maintained within an appropriate temperature range according to its intended use and may include a "refrigeration room," "freezing room," or "variable temperature room" distinguished according to its intended use and / or temperature range. The refrigerator room may be maintained at a temperature appropriate for refrigerated storage of goods, and the freezer room may be maintained at a temperature appropriate for frozen storage of goods. "Refrigeration" may mean cooling goods to a temperature that does not freeze them; for example, the refrigerator room may be maintained within a range of 0°C to 7°C. "Freezing" may mean cooling goods to freeze them or to maintain them in a frozen state; for example, the freezer room may be maintained within a range of -20°C to -1°C. The variable temperature room may be used as either a refrigerator room or a freezer room, with or without the user's choice.

[0047] Storage rooms may be referred to by various names, such as "vegetable room," "fresh room," "cooling room," and "ice-making room," in addition to terms like "refrigeration room," "freezing room," and "variable temperature room." The terms "refrigeration room," "freezing room," and "variable temperature room" used below should be understood as encompassing storage rooms with corresponding uses and temperature ranges.

[0048] According to one embodiment, the refrigerator may include at least one door configured to open and close one side of the storage compartment. The door may be provided to open and close each of one or more storage compartments, or a single door may be provided to open and close multiple storage compartments. The door may be installed to be rotatable or sliding on the front of the main body.

[0049] The “door” may be configured to seal the storage room when the door is closed. The door may include insulation material, similar to the main body, to insulate the storage room when the door is closed.

[0050] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the rear of the door and facing the storage room, an upper cap, a lower cap, and a door insulation material provided inside the same.

[0051] A gasket may be provided on the edge of the door inner panel to seal the storage compartment by adhering to the front of the main body when the door is closed. The door inner panel may include a dyke that protrudes rearward to allow a door basket for storing items to be mounted.

[0052] According to one embodiment, the door may include a door body and a front panel detachably coupled to the front side of the door body and forming the front of the door. The door body may include a door outer panel forming the front of the door body, a door inner panel forming the rear of the door body and facing the storage compartment, an upper cap, a lower cap, and a door insulation material provided inside them.

[0053] Refrigerators can be classified into French Door Type, Side-by-side Type, BMF (Bottom Mounted Freezer), TMF (Top Mounted Freezer), or 1-door refrigerators depending on the arrangement of the door and storage compartment.

[0054] According to one embodiment, the refrigerator may include a cold air supply device arranged to supply cold air to the storage compartment.

[0055] The "cold air supply device" may include a machine, apparatus, electronic device, and / or a system combining these that can generate cold air and guide cold air to cool a storage room.

[0056] According to one embodiment, a cold supply device can generate cold air through a refrigeration cycle that includes the processes of compression, condensation, expansion, and evaporation of a refrigerant. To this end, the cold supply device may include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. According to one embodiment, the cold supply device may include a semiconductor such as a thermoelectric element. The thermoelectric element can cool a storage chamber through heat generation and cooling action via the Peltier effect.

[0057] According to one embodiment, the refrigerator may include a machine room arranged to accommodate at least some parts belonging to a cold air supply device.

[0058] The "machine room" may be configured to be partitioned and insulated from the storage room to prevent heat generated from components placed in the machine room from being transferred to the storage room. The interior of the machine room may be configured to communicate with the exterior of the main body to dissipate heat from components placed inside the machine room.

[0059] According to one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without opening the door.

[0060] According to one embodiment, the refrigerator may include an ice-making device configured to generate ice. The ice-making device may include an ice-making tray that stores water, an ice-removing device that separates ice from the ice-making tray, and an ice bucket that stores the ice generated from the ice-making tray.

[0061] According to one embodiment, the refrigerator may include a control unit for controlling the refrigerator.

[0062] The "control unit" may include a memory that stores or remembers a program and / or data for controlling a refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc., according to the program and / or data stored in the memory.

[0063] The memory stores or records various information, data, commands, programs, etc., necessary for the operation of the refrigerator. The memory can store temporary data generated while generating control signals to control the components included in the refrigerator. The memory may include at least one of volatile memory or non-volatile memory, or a combination thereof.

[0064] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operation of an artificial intelligence model. Additionally, the processor may include a central processing unit, a graphics processing unit (GPU), etc. The processor can generate control signals to control the operation of the cold air supply unit. For example, the processor can receive temperature information of the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cold air supply unit based on the temperature information of the storage compartment.

[0065] Additionally, the processor can process user input of the user interface and control the operation of the user interface according to programs and / or data stored in memory. The user interface may be provided using an input interface and an output interface. The processor can receive user input from the user interface. Additionally, the processor can transmit display control signals and image data to the user interface to display an image on the user interface in response to the user input.

[0066] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one sub-processor. The memory may include one or more memory units.

[0067] According to one embodiment, the refrigerator may include a processor and memory that control all components included in the refrigerator, and may include a plurality of processors and a plurality of memories that individually control the components of the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cold air supply device according to the output of a temperature sensor. Additionally, the refrigerator may separately provide a processor and memory that control the operation of a user interface according to user input.

[0068] The communication module can communicate with external devices, such as servers, mobile devices, and other home appliances, through nearby Access Points (APs). The Access Point (AP) can connect the Local Area Network (LAN) to which the refrigerator or user device is connected to the Wide Area Network (WAN) to which the server is connected. The refrigerator or user device can be connected to the server through the Wide Area Network (WAN).

[0069] The input interface may include keys, touchscreens, microphones, etc. The input interface may receive user input and transmit it to the processor.

[0070] The output interface may include a display, a speaker, etc. The output interface can output various notifications, messages, information, etc. generated by the processor.

[0071] Refrigerators according to various embodiments will be described in detail below with reference to the attached drawings.

[0072] FIG. 1 is a drawing for explaining the operation of a refrigerator according to at least one embodiment of the present disclosure.

[0073] Referring to FIG. 1, a user (1) can place an object into a refrigerator (100). In the present disclosure, the object may include food (e.g., milk).

[0074] The refrigerator (100) may include a camera (110) on the top of the main body (10).

[0075] The refrigerator (100) can acquire an image of an object entering the refrigerator (100) using a camera (110). The refrigerator (100) can identify the type of object entering the refrigerator (100) based on the acquired image. Additionally, the refrigerator (100) can identify the type and size of the identified object and provide a notification to the user (1) regarding the optimal storage location of the object entering the refrigerator (e.g., “Please put the milk in the right compartment of the third shelf”) based on the identified type and size.

[0076] Additionally, the refrigerator (100) can identify the location where an object finally arrives within the storage area inside the refrigerator and update the database for the storage area inside the refrigerator. The database may include information regarding the location of objects and other food items stored in the storage area, and the remaining space in each area.

[0077] A specific method for the refrigerator (100) to identify the optimal storage location of an object and to update the database for the storage area is described later based on FIG. 5.

[0078] FIG. 2 is a drawing for explaining the exterior of a refrigerator according to at least one embodiment of the present disclosure.

[0079] Referring to FIG. 2, the refrigerator (100) may include a main body (10) having a storage room (21, 22) formed therein and a door (31, 32, 41, 42) that opens and closes one side of the storage room (21, 22).

[0080] The main body (10) may include an outer body and an inner body provided on the inner side of the outer body to form storage rooms (21, 22). The main body (10) may include an insulating material provided between the outer body and the inner body. The main body (10) may include a partition wall (11) that divides the storage rooms (21, 22) vertically.

[0081] The storage rooms (21, 22) may include a refrigerator room for refrigerating items and a freezer room for freezing items. For example, the storage room (21) provided on the upper part of the main body (10) may be used as a refrigerator room (21), and the storage room (22) provided on the upper part of the main body (10) may be used as a freezer room (22).

[0082] The storage room (21, 22) is provided with an open front so that items can be taken out and put in, and the open front of the storage room (21, 22) can be opened and closed by a door (31, 32, 41, 42). A handle may be provided on the door (31, 32, 41, 42).

[0083] The doors (31, 32, 41, 42) may include a refrigerator door (31, 32) for opening and closing the refrigerator room (21) and a freezer door (41, 42) for opening and closing the freezer room (22).

[0084] The refrigerator door (31, 32) can be connected to the main body (10) so as to be rotatable in the left and right directions. A door guard (33) for storing items can be provided on the back of the refrigerator door (31, 32). The refrigerator (21) can be provided with a shelf (23) for placing items and a drawer-type storage container (24) for storing items.

[0085] Referring to FIG. 2, the refrigerator (100) may include a plurality of doors. The plurality of doors may include an upper French door (31, 32), a middle drawer door (41), and a lower drawer door (42). The middle drawer door (41) and the basket (43) connected to the middle drawer door (41) may be referred to as the middle drawer, and the lower drawer door (42) and the basket (44) connected to the lower drawer door (42) may be referred to as the lower drawer. In the following description, the operation of the refrigerator (100) is described on the premise that the upper French door (31, 32) is open. However, the operation according to the present disclosure is not limited to the case where the upper French door (31, 32) is open, and it is obvious that the operation according to the present disclosure can also be performed when the middle drawer door (41) and the lower drawer door (42) are open.

[0086] FIG. 3 is a block diagram illustrating the configuration of a refrigerator according to at least one embodiment of the present disclosure.

[0087] Referring to FIG. 3, a refrigerator (100) according to the present disclosure may include a camera (110), a memory (120), and at least one processor (130) (hereinafter referred to as processor (130)).

[0088] The camera (110) can generate images. The camera (110) is configured to capture objects and storage areas inside the refrigerator. Therefore, the images captured by the camera (110) can capture food stored in the storage areas inside the refrigerator (100) and food being put in and taken out. The camera (110) can be placed at various locations on the refrigerator (100). For example, referring to FIG. 2, the camera (110) can be located at the top front of the main body (10). Thus, the camera (110) can be exposed to the outside when the doors (31, 32) are opened. The location where the camera (110) is attached is not limited to this and can be attached at various locations on the main body (10).

[0089] The camera (110) may include a lens assembly and an image sensor. However, such configuration is exemplary, and it is understood that new configurations may be added or some configurations omitted in addition to such configurations when carrying out the present disclosure. For example, the camera (110) may further include an image signal processor (ISP). The image signal processor may be configured as at least part of the processor (130).

[0090] The camera (110) may include an RGB camera. The RGB camera may include a camera that outputs an image in which an image acquired using the camera is recorded with three basic generation information of red, green, and blue.

[0091] A lens assembly included in the camera (110) can collect light incident from the outside. The lens assembly may include one or more lenses. For example, the lens assembly can refract light incident from the outside. The refracted light can be collected on an image sensor.

[0092] An image sensor can generate an image corresponding to light received through a lens assembly. The image sensor can be implemented as a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. The image sensor may include a pixel array composed of multiple pixels. For example, the image sensor can generate an image by converting light into an electrical image signal using multiple pixels. The image signal can be amplified and converted into a digital signal, after which it can be processed. The image generated by the image sensor can be provided to a processor (130).

[0093] According to various embodiments of the present disclosure, the memory (120) may store data necessary for the refrigerator (100) to operate. Depending on the purpose of data storage, the memory (120) may be implemented as a memory embedded in the refrigerator (100) (e.g., volatile memory (e.g., semi-permanent memory such as RAM (random access memory)), non-volatile memory (e.g., permanent memory such as ROM (read-only memory)), flash memory, hard drive or solid-state drive, etc.), or as a memory that can be attached to the refrigerator (100) (e.g., memory card, external memory, etc.).

[0094] Instructions may be stored in the memory (120). The processor (130) may perform the operation of the refrigerator (100) according to various embodiments of the present disclosure by executing the instructions in the memory (110) individually or collectively. Additionally, programs and data for operating the refrigerator (100) may be stored in the memory (120). For example, the memory (120) may store one or more software applications, such as operating system (or system) software applications, firmware software applications, driver software applications, plugin (e.g., add-in, add-on, and / or applet) software applications, and / or any other suitable software applications.

[0095] Meanwhile, in the present disclosure, the term memory (120) may be used to include memory (120), ROM, RAM within the processor (130), or a memory card (e.g., micro SD card, memory stick) mounted in the refrigerator (100). In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (120), and the information stored in the memory (120) may be updated as it is received from an external device or input by a user.

[0096] The processor (130) can control the overall operations of the refrigerator (100). For example, the processor (130) can cause other components of the electronic device (100) to perform various operations by executing instructions stored in memory (120). For example, the processor (130) can control the operation of the refrigerator (100) by being operatively connected to memory (120). Additionally, the processor (130) can control the operation of the refrigerator (100) according to the present disclosure by executing one or more instructions stored in memory (120). The processor (130) may be composed of one or more processors.

[0097] The processor (130) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing operations. The processor (130) may include at least one electrical circuit and may process instructions (or programs, data, etc.) stored in memory (120) individually or collectively. The processor (130) may include a processor assembly comprising one or more processing circuits. The processor (130) may include any processing circuit that is operative to control the performance and operations of one or more components (e.g., memory (110)) of the electronic device (100). For example, the processor (130) (e.g., application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or chipset). For example, the processor (130) may be implemented with a plurality of cores (or at least one core circuit), a plurality of chips, or a plurality of chipsets. For example, the processor (130) may include one or more processing circuits. For example, the processor (130) may include one or more processing circuits configured to perform the various functions of the present disclosure individually and / or collectively.

[0098] The processor (130) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. The processor (130) may control one or any combination of other components of the refrigerator (100) and may perform operations or data processing related to communication. The processor (130) may execute one or more programs or instructions stored in the memory (120) of the refrigerator (100). For example, the processor (130) may perform a method according to one embodiment of the present disclosure by executing one or more instructions stored in the memory (120).

[0099] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0100] The processor (130) may be implemented as a single-core processor including one core, or as one or more multi-core processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). When the processor (120) is implemented as a multi-core processor, each of the multiple cores included in the multi-core processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multi-core processor. Additionally, each of the multiple cores included in the multi-core processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0101] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0102] In the embodiments of the present disclosure, a processor may mean a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto.

[0103] FIG. 4 is a block diagram illustrating the detailed configuration of a refrigerator according to at least one embodiment of the present disclosure.

[0104] Referring to FIG. 4, the refrigerator (100) may include a camera (110), a memory (120), a processor (130), a sensor (140), a communication circuit (150), an input interface (160), and an output interface (170).

[0105] However, such configurations are exemplary, and it is understood that new configurations may be added or some configurations omitted in addition to such configurations when implementing the present disclosure. Meanwhile, detailed descriptions of configurations shown in FIG. 4 that overlap with configurations shown in FIG. 3 will be omitted.

[0106] The sensor (140) may include a configuration that detects when the door is open.

[0107] The sensor (140) may include a lead sensor. Additionally, the sensor (140) may be included in the main body (10) of the refrigerator (100).

[0108] A reed sensor may include two metal leads. The metal leads respond to a magnetic field and may come into contact with or separate from each other. For example, the metal leads may come into contact if a magnet is present around the reed sensor. Conversely, the metal leads may remain separated if no magnet is present around the reed sensor.

[0109] The sensor (140) can detect the opening of the door by utilizing the properties of the metal leads. For example, the door of the refrigerator (100) may have a magnet built into it. When the door of the refrigerator (100) is closed, two metal leads included in the sensor (140) may come into contact with each other. When the metal leads come into contact with each other, current may flow through the circuit containing the sensor (140). The refrigerator (100) can identify that the door is closed when current flows through the circuit containing the sensor (140).

[0110] On the other hand, when the door of the refrigerator (100) is opened, the two metal leads included in the sensor (140) may be separated from each other. When the metal leads are separated from each other, current may not flow through the circuit containing the sensor (140). The refrigerator (100) can identify that the door is open if current does not flow through the circuit containing the sensor (140).

[0111] The communication circuit (150) can perform data communication with an external electronic device under the control of the processor (130). The external electronic device may include a server, home appliance, mobile device (e.g., smartphone, tablet PC, wearable device, etc.). The communication circuit (150) can perform data communication with an external electronic device (e.g., server, home appliance, mobile device, etc.) under the control of the processor (130). For example, the communication circuit (150) can communicate with an external electronic device through a network. For example, the processor (130) can receive data from an external electronic device through the communication circuit (150) and transmit data to an external electronic device through the communication circuit (150).

[0112] The communication circuit (150) may include hardware components to support the transmission and / or reception of electrical signals between the refrigerator (100) and an external electronic device. For example, the communication circuit (150) may perform data communication between the refrigerator (100) and the electronic device using at least one of a data communication method including wired LAN, wireless LAN, Wi-Fi, Wi-Fi Direct, Bluetooth, ZigBee, WFD (Wi-Fi Direct), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet), WiMAX (World Interoperability for Microwave Access), SWAP (Shared Wireless Access Protocol), WiGig (Wireless Gigabit Alliances), and RF communication. The communication circuit (150) may also be expressed as a communication interface.

[0113] The input interface (160) includes circuitry. The input interface (160) can receive user input and transmit the user input to the processor (130). For example, the input interface (160) can receive various user inputs for setting or selecting various functions supported by the refrigerator (100).

[0114] The input interface (160) may include various types of input devices.

[0115] According to one example, the input interface (160) may include a physical button. The physical button may include a function key or a dial button. The physical button may be implemented as one or more keys.

[0116] According to one example, the input interface (160) can receive user input using a touch method. For example, the input interface (160) can be implemented as a touch screen capable of performing the function of a display (171).

[0117] According to one example, the input interface (160) can receive user voice using a microphone. The processor (130) can perform a function corresponding to the user voice using voice recognition. For example, the processor (130) can convert the user voice into text data using a Speech To Text (STT) function, obtain control command data based on the text data, and perform a function corresponding to the user voice based on the control command data. According to an embodiment, the STT function may be performed on an external server.

[0118] The output interface (170) may include a display (171) and a speaker (172).

[0119] The display (171) can output visualized information to the user. The visualized information may include visual objects displayed on the screen of the display (171). For example, the visual objects may include images, UI (user interface) elements, etc. The UI elements may include icons, GUI (graphic user interface), etc.

[0120] The display (171) may be implemented as a display including a self-emissive element or as a display including a non-emissive element and a backlight. For example, the display (171) may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes) display, a micro LED display, a Mini LED display, a QLED (Quantum dot light-emitting diodes) display, etc.

[0121] The speaker (172) can output an audio signal. The processor (130) can output warning sounds, notification messages, response messages corresponding to user input, etc. related to the operation of the refrigerator (100) through the speaker (172).

[0122] An LED (light emitting diode, 173) can emit light. The LED (173) may include a semiconductor device that converts electrical energy into light. A processor (130) may attach an LED to each of a plurality of storage compartments. When the processor (130) provides a notification related to the operation of the refrigerator (100), it may cause the LED attached to the storage compartment related to the notification among the LEDs attached to each of the plurality of storage compartments to emit light.

[0123] FIG. 5 is a flowchart for explaining the operation of a refrigerator according to at least one embodiment of the present disclosure. A processor (130) may perform at least one of the operations of FIG. 5. Instructions stored in memory (120) may cause the refrigerator (100) to perform the operations of FIG. 5 when executed individually or collectively by the processor (130).

[0124] In operation 510, the refrigerator (100) can identify that the refrigerator door is open. The refrigerator (100) can identify that the refrigerator door is open based on a sensor (140) included in the refrigerator door.

[0125] The sensor (140) may include a lid sensor. The lid sensor may be a sensor that detects the opening or closing of the door of the refrigerator (100). The specific operation of the lid sensor detecting the opening or closing of the door of the refrigerator (100) is the same as described above in FIG. 4.

[0126] For example, if the refrigerator (100) identifies that no current is flowing through the circuit containing the lead sensor, it can identify that the door has been opened.

[0127] The method by which the refrigerator (100) detects door opening is not limited to using a lid sensor, and door opening can be detected through various methods.

[0128] In operation 515, the refrigerator (100) can acquire a first image through the camera (110).

[0129] When the refrigerator (100) identifies that the door is open, it can acquire a first image using a camera (110). The camera (110) may be mounted in a fixed form on the outside, mounted in the inside, mounted with multiple cameras on both the inside and outside, or installed movably on the refrigerator so that the camera (110) can protrude outward from the refrigerator (100). In the following description, the camera (110) is described as being installed movably on the refrigerator so that the camera (110) can protrude outward from the refrigerator (100).

[0130] Since the camera (110) is located at the top of the front part of the main body (10), the refrigerator (100) can obtain an image taken from the top part of the front of the refrigerator (100) in a downward direction.

[0131] The first image may include an image capturing the process of the door of the refrigerator (100) opening and an object being placed into the refrigerator. Accordingly, the first image may include an image capturing the exterior of the object and a portion of the multiple storage areas included in the refrigerator.

[0132] In addition, the image may include a video.

[0133] In operation 520, the refrigerator (100) can identify the type and size of an object being put into the refrigerator based on the first image.

[0134] When the refrigerator (100) acquires a first image, it can identify an object included in the first image. The refrigerator can extract a key frame from the first image. A key frame may be a frame containing key information in the image. For example, the refrigerator (100) can identify the average of pixel values ​​between frames arranged in a time series. A pixel value is a value for each pixel and may include color and brightness information of the pixel. The refrigerator (100) can identify the average of pixel values ​​of multiple pixels included in a frame.

[0135] Additionally, the refrigerator (100) can identify frames with a large change in the average among all frames. The refrigerator (100) can identify a frame acquired close to the current time point among frames with a large change as a key frame.

[0136] The refrigerator (100) can identify objects included in the first image based on the extracted key frame.

[0137] The refrigerator can identify the type, coordinates, and size of an object included in the key frame of the first image using an artificial intelligence model. The artificial intelligence model can be stored in memory (120). The artificial intelligence model may have data for food corresponding to multiple types pre-learned.

[0138] Accordingly, the refrigerator (100) can identify the type of object by inputting the key frame of the first image into an artificial intelligence model. For example, assume a situation where a user (1) puts milk into the refrigerator (100). When the refrigerator (100) inputs the key frame of the first image into the artificial intelligence model, it can identify that the type of object included in the first image is milk.

[0139] In addition, the refrigerator (100) can input the key frame of the first image into an artificial intelligence model to identify the coordinates of the object.

[0140] In one embodiment, the artificial intelligence model can identify the coordinates of an input image using a three-dimensional coordinate system. The artificial intelligence model may be a neural network model trained to estimate the three-dimensional position of an object from an image.

[0141] Therefore, when the refrigerator (100) inputs the key frame of the first image into the artificial intelligence model, it can obtain the x, y, and z coordinates of the object.

[0142] The refrigerator (100) can identify the size of an object based on the coordinates of the object obtained from an artificial intelligence model. The size may include the length and volume for the width, height, and depth of the object.

[0143] A specific method for the refrigerator (100) to identify the size of an object is described later based on operation 555.

[0144] The refrigerator (100) can identify the volume of an object based on identifying the width, length, and height of the object.

[0145] In operation 525, the refrigerator (100) can identify a first storage area corresponding to the object among a plurality of storage areas.

[0146] The storage compartment of the refrigerator (100) may include a plurality of storage areas. The plurality of storage areas may include a plurality of shelves and a plurality of storage cabinets. Additionally, the first storage area may be the optimal storage area for an object being received among the plurality of storage areas.

[0147] The refrigerator (100) can use data stored in memory (120) to identify the first storage area.

[0148] For example, data regarding multiple storage areas may be stored in the memory (120) of the refrigerator (100). The data regarding multiple storage areas may include data regarding the type of food stored in each of the multiple storage areas and the size of the remaining space. The size of the remaining space may be the volume of the remaining space. Additionally, the size of the remaining space may include information regarding the remaining area of ​​the space floor and the remaining height of the space. Information regarding the size of the remaining space of the multiple storage compartments may be updated and stored in the memory (120) whenever an object is put in or taken out. Specific details regarding this will be described later.

[0149] The refrigerator (100) can identify a first storage area among a plurality of storage areas based on identifying the type and size of the object.

[0150] For example, if a refrigerator identifies that a dedicated storage area exists for an object, it can identify the remaining space of the dedicated storage area.

[0151] The dedicated storage area may be an area for storing food of a preset type. The dedicated storage area may include, for example, a storage area operating in a vegetable-only mode, a storage area operating in a fish-only mode, and a storage area operating in a wine-only mode.

[0152] For example, if the refrigerator (100) identifies that the type of object being received is a vegetable, it can identify the size of the remaining space of the storage area that operates in a vegetable-only mode among the multiple storage areas.

[0153] The refrigerator (100) can identify the size of the remaining space in the dedicated storage area based on data regarding the size of the object stored in the dedicated storage area and the size of the dedicated storage area stored in the memory (120).

[0154] When the refrigerator (100) identifies the size of the remaining space of the dedicated storage area, it can identify whether an object can be placed in the remaining space of the dedicated storage area based on the size of the object being placed. When the refrigerator (100) identifies that an object can be placed in the remaining space of the dedicated storage area, it can identify the dedicated storage area as the first storage area.

[0155] In one embodiment, the refrigerator (100) can identify that an object being received can be stored in the dedicated storage area if the size of the remaining space in the dedicated storage area is greater than or equal to the size of the object being received. On the other hand, if the size of the remaining space in the dedicated storage area is less than the size of the object being received, the refrigerator can identify that an object being received cannot be stored in the dedicated storage area.

[0156] For example, the refrigerator (100) can determine whether an object can be stored in the dedicated storage area by comparing the remaining area of ​​the dedicated storage area with the area of ​​the object. The refrigerator (100) can determine that an object can be stored in the dedicated storage area if the remaining area of ​​the dedicated storage area is greater than or equal to the remaining area of ​​the object.

[0157] On the other hand, the refrigerator (100) can identify whether an object can be stored in the dedicated storage area based on the remaining height of the dedicated storage area if the remaining area of ​​the dedicated storage area is less than the remaining area of ​​the object.

[0158] For example, the refrigerator (100) can identify that an object can be stored in the dedicated storage area if the remaining area of ​​the dedicated storage area is less than the remaining area of ​​the object, but the remaining height of the dedicated storage area is greater than the height of the object.

[0159] On the other hand, the refrigerator (100) can identify that an object cannot be stored in the dedicated storage area if the remaining area of ​​the dedicated storage area is less than the remaining area of ​​the object and the remaining height of the dedicated storage area is smaller than the height of the object. If the refrigerator (100) identifies that there is no dedicated storage area or that an incoming object cannot be stored in the dedicated storage area, it can identify a storage area in which an object of the same type as the object is stored.

[0160] For example, the refrigerator (100) can identify the type of object stored in each of the multiple storage areas based on data stored in memory (120). The refrigerator (100) can identify the storage area among the multiple storage areas where an object of the same type as the object is stored.

[0161] For example, it is assumed that the object being received is cola. The refrigerator (100) can identify the storage area where cola is stored among the multiple storage areas based on the data stored in the memory (120).

[0162] When the refrigerator (100) identifies a storage area where an object of the same type as the object is stored, it can identify the size of the remaining space in the identified storage area.

[0163] Based on identifying the size of the remaining space of the identified storage area, the refrigerator (100) can identify whether an incoming object can be placed in the remaining space of the identified storage area. If the refrigerator (100) identifies that an incoming object can be placed in the remaining space of the identified storage area, it can identify the storage area that has a history of storing food of the same type as the object as the first storage area.

[0164] If the refrigerator (100) identifies that an object being entered cannot be entered into the remaining space of the identified storage area, or that there is no storage area where an object of the same type as the object is stored, it can identify a storage area among a plurality of storage areas that has a history of storing food of the same type as the object based on the type of the object being entered.

[0165] For example, it is assumed that the type of object being placed in the refrigerator (100) is cola. Based on the data stored in the memory (120), the refrigerator (100) can identify a storage area among multiple areas where there is a history of storing cola.

[0166] When the refrigerator (100) identifies a storage area that has a history of storing food of the same type as the object, it can identify the size of the remaining space of the identified storage area.

[0167] The refrigerator (100) can determine whether an object being received can be stored in the remaining space of the identified area based on the size of the remaining space of the identified storage area. For example, the refrigerator (100) can determine whether the object can be stored in the storage area where food of the same type as the object has been stored by comparing the remaining area of ​​the storage area where food of the same type as the object has been stored with the area of ​​the object. If the area of ​​the storage area where food of the same type as the object has been stored is greater than or equal to the remaining area of ​​the object, the refrigerator (100) can determine that the object can be stored in the storage area where food of the same type as the object has been stored.

[0168] On the other hand, if the storage area where food of the same type as the object has been stored is less than the remaining area of ​​the object, the refrigerator (100) can identify whether the object can be stored in the storage area where food of the same type as the object has been stored based on the remaining height of the storage area where food of the same type as the object has been stored.

[0169] For example, if the storage area where food of the same type as the object has been stored is less than the remaining area of ​​the object, but the remaining height of the storage area where food of the same type as the object has been stored is greater than the height of the object, it can be identified that the object can be stored in the storage area where food of the same type as the object has been stored.

[0170] On the other hand, the refrigerator (100) can identify that the object cannot be stored in the storage area where food of the same type as the object has been stored, if the storage area where food of the same type as the object has been stored is less than the remaining area of ​​the object and the remaining height of the storage area where food of the same type as the object has been stored is smaller than the height of the object.

[0171] If the refrigerator (100) identifies whether an object being entered can be entered into the remaining space of the identified area, it can identify the storage area that has a history of storing food of the same type as the object as the first storage area.

[0172] Although the refrigerator (100) has performed the aforementioned operation, there may be cases where it fails to identify the location of the first storage area where the object is to be stored. Based on the size of the object, the refrigerator (100) can identify the area among the multiple storage areas where there is remaining space for the object to be placed as the first storage area.

[0173] In operation 530, the refrigerator (100) can provide a first notification to the user to guide the first storage area.

[0174] When the refrigerator (100) identifies that an object is being placed in the refrigerator (100), it may provide a notification to the user to store the object in the first storage area. The notification may include, for example, a voice notification, a cabinet pop-up notification, a light notification, and a display notification.

[0175] The voice notification may include a method in which the refrigerator (100) outputs a notification to store an object in the first storage area through a speaker (172) built into the refrigerator (100) (e.g., outputting a voice notification such as “Please put the milk in the right compartment of the third layer” through the speaker (172)).

[0176] The pop-up notification may include a method of protruding the cabinet and shelf corresponding to the first storage area based on the entry of an object when the storage area corresponding to the first storage area of ​​the refrigerator (100) is a cabinet and shelf. A motor may be included at the rear of the cabinet so that the refrigerator (100) protrudes the cabinet. The configuration in which the refrigerator (100) performs the pop-up notification is not limited to cabinets and shelves, and the refrigerator (100) can perform pop-up notifications for various configurations.

[0177] The lighting notification may include a method in which the refrigerator (100) turns on the lighting included in the storage area corresponding to the first storage area.

[0178] Additionally, the display notification may include a method of displaying a notification for storing an object in the first storage area on a display (171) built into the refrigerator (100) (e.g., displaying a notification such as “Please put the milk in the right compartment of the third layer” on the display (171)).

[0179] In operation 535, the refrigerator (100) can identify a second storage area in which an object is received among a plurality of storage areas based on the first image. The second storage area can identify an area in which an object received by the refrigerator (100) is received among a plurality of storage areas.

[0180] Accordingly, based on the first image, the refrigerator (100) can identify the area where an object entering the refrigerator (100) is entered among a plurality of storage areas.

[0181] For example, referring to FIG. 6, a plurality of storage areas may include each space (601 to 608) in which the storage compartment of the refrigerator (100) is partitioned. As described above, the plurality of storage areas may include a plurality of shelves (601 to 606) and a plurality of storage cabinets (607, 608).

[0182] Additionally, when the opening of the doors (601, 602) of the refrigerator (100) is identified, images of multiple storage areas can be obtained using a camera (110) attached to the top of the front part of the main body.

[0183] Based on the first image obtained, the refrigerator (100) can identify the number of layers of the storage area into which an object is placed. In the following description, among the multiple storage areas included in the storage room of the refrigerator (100), the storage area located at the bottom can be identified as the first layer, and among the storage areas adjacent to the first layer, the storage area located on a layer higher than the first layer can be identified as the second layer. In the following description, the storage room is described as having two layers, but the present disclosure is not limited thereto, and the storage room may include storage areas of various layers.

[0184] The refrigerator (100) can acquire a first image using a camera (110) and detect multiple edges corresponding to multiple storage areas in the acquired image.

[0185] Referring to FIG. 7, the refrigerator (100) can identify an area within a preset range in a first image obtained using a camera (110) as a region of interest (710). The region of interest (710) may be an area including a storage room, a boundary portion between the refrigerator (100) and the outside area, and a preset range among the outside area of ​​the storage room.

[0186] The refrigerator (100) may not identify lines for backgrounds and objects that exist outside a preset range in the first image obtained. Accordingly, the refrigerator (100) identifies only lines that exist within a preset range in the image, thereby more accurately identifying multiple edges corresponding to multiple storage areas.

[0187] The refrigerator (100) can preprocess key frames among the acquired first images to remove noise and minimize contrast within the image. The refrigerator (100) can identify edges inside the refrigerator based on the preprocessed image.

[0188] The refrigerator (100) can identify edges inside the refrigerator using an edge detector and a Hough transform in a preprocessed image.

[0189] An edge detector may include an algorithm for identifying boundary regions in an acquired image. For example, the refrigerator (100) can identify boundary lines based on changes in pixel values ​​included in a preprocessed image. The refrigerator (100) can identify pixel values ​​corresponding to each pixel included in the image and identify differences in pixel values ​​between adjacent pixels. The refrigerator (100) can identify two pixels among the pixels included in the image where the difference in adjacent pixel values ​​is greater than or equal to a preset value. The refrigerator (100) can identify the pixel with the larger pixel value among the two pixels as the pixel included in the edge.

[0190] Additionally, when the refrigerator (100) identifies pixels corresponding to edges in an image, it can identify multiple edges corresponding to multiple storage areas using a Hough transform.

[0191] In addition, the Hough transform may include an algorithm capable of identifying edges corresponding to preset conditions based on edges identified using an edge detector.

[0192] For example, the refrigerator (100) can identify an edge corresponding to a preset condition among a plurality of edges. The preset condition may include, for example, an edge whose identified length is greater than or equal to a preset value and an edge included in a specific area among the areas included in the first image.

[0193] Accordingly, the refrigerator (100) can identify an edge among the identified edges that corresponds to a preset condition. The refrigerator (100) can identify the identified edge as a plurality of edges corresponding to a plurality of storage areas.

[0194] The method by which the refrigerator (100) identifies the edges of multiple storage areas is not limited to using the aforementioned edge detector and Hough transform, and the edges can be identified using various methods.

[0195] The refrigerator (100) can identify a storage area where an object (80) is received among a plurality of storage areas based on a plurality of edges.

[0196] The refrigerator (100) can identify a storage area where an object (80) is placed using a first image obtained. At this time, the first image may include a storage room (800) of the refrigerator (100) and a plurality of edges (810 to 830).

[0197] For example, the refrigerator (100) can identify the number of layers of storage areas in which an object (80) is placed in the refrigerator among a plurality of storage areas based on a plurality of edges.

[0198] For example, referring to FIG. 8, a plurality of edges may include an edge (810) for identifying the receipt or receipt of an object, an edge (820) corresponding to a storage area located on the first floor among the plurality of storage areas, and an edge (830) corresponding to a storage area located on the second floor among the plurality of storage areas.

[0199] When the refrigerator (100) identifies that the object (80) has been received or released, it can identify that the object (80) has been received or released when the object (80) has been identified as having come into contact with the edge (810) for identifying the receipt or release of the object (80).

[0200] If the refrigerator (100) identifies that the direction of movement of the object is from the outside to the inside of the refrigerator by comparing the frames included in the first image, it can identify that the object (80) is being entered into the refrigerator (100). On the other hand, if the refrigerator (100) identifies that the direction of movement of the object (80) is from the inside to the outside of the refrigerator by comparing the frames included in the first image, it can identify that the object (80) is being taken out of the refrigerator (100).

[0201] The refrigerator (100) can identify the direction of movement of the object (80) by comparing the y-coordinate of the identified edge with the y-coordinate of the object. For example, if the refrigerator (100) has a value greater than the y-coordinate of the object identified in the frame acquired at t seconds, but has a value smaller than the y-coordinate of the object identified in the frame acquired at t+x seconds, the direction of movement of the object (80) can be identified as moving from the outside of the refrigerator to the inside. In this case, the refrigerator (100) can identify that the object (80) is being entered.

[0202] On the other hand, if the y-coordinate of the object (80) has a smaller value than the y-coordinate of the edge identified in the frame acquired at t seconds, but the y-coordinate of the object has a larger value than the y-coordinate of the edge identified in the frame acquired at t+x seconds, the direction of movement of the object (80) can be identified as moving from inside the refrigerator to outside. In this case, the refrigerator (100) can identify that the object (80) is being released.

[0203] When the refrigerator (100) identifies that the object (80) has come into contact with the edge (820) corresponding to the storage area located on the first floor among the multiple storage areas, it can identify that the object (80) is being put into the storage area located on the first floor.

[0204] Additionally, the refrigerator (100) can identify that the object (80) is being placed in the storage area located on the second floor when the object (80) comes into contact with the edge (820) corresponding to the storage area located on the first floor among the multiple storage areas and the edge (830) corresponding to the storage area located on the second floor among the multiple storage areas.

[0205] For example, if the refrigerator (100) identifies that the milk has come into contact with the edge (820) corresponding to the storage area located on the first floor but has not come into contact with the edge (830) corresponding to the storage area located on the second floor among the multiple storage areas, it can be identified that the milk has come into contact with the first floor among the multiple storage areas.

[0206] Additionally, if the refrigerator (100) identifies that the milk is in contact with the edge (820) corresponding to the storage area located on the first floor and the edge (830) corresponding to the storage area located on the second floor among the multiple storage areas, it can identify that the milk is being stored on the second floor among the multiple storage areas.

[0207] The refrigerator (100) can identify whether an object (80) is being stored in the right area or the left area in the layer of the identified storage area.

[0208] For example, the refrigerator (100) can identify the storage area where the object (80) is stored by comparing the x-coordinate of the coordinates of the object (80) included in the first image and the coordinates corresponding to the center of the first image.

[0209] For example, if the refrigerator (100) identifies that the x-coordinate of the identified object (80) has a smaller value than the x-coordinate corresponding to the center of the first image, it can identify that the object (80) is placed in the right area of ​​the layer of the identified storage area. On the other hand, if the refrigerator (100) identifies that the x-coordinate of the identified object (80) has a larger value than the x-coordinate corresponding to the center of the first image, it can identify that the object (80) is placed in the left area of ​​the layer of the identified storage area.

[0210] Although the above example is written on the premise that an object (80) is received, the present disclosure is not limited thereto, and even when an object (80) is released, the camera can be adjusted by identifying the area where the user (1)’s hand is directed among the multiple storage areas.

[0211] Additionally, the method by which the refrigerator (100) identifies whether an object (80) has been received is not limited to the examples described above. In one embodiment, the refrigerator (100) may identify whether an object (80) has been received using an artificial intelligence model. The artificial intelligence model may be stored in memory (120).

[0212] The artificial intelligence model may be in a state where learning is performed by receiving inputs such as an image of the object (80) in contact with an edge, an image of the object (80) not in contact with an edge, and an image of the storage room in which the object (80) is received.

[0213] The refrigerator (100) can obtain the location of the storage room where the object is received by inputting an image of the object being received into an artificial intelligence model. Additionally, the refrigerator (100) can obtain the location of the storage room where the object is released by inputting an image of the object being released into an artificial intelligence model.

[0214] The aforementioned artificial intelligence model can be replaced with terms such as an inbound / outbound status classification model.

[0215] In operation 540, the refrigerator (100) can identify the protruding length of the camera (110) based on the second storage area.

[0216] Based on identifying the second storage area, the refrigerator (100) can identify the protruding length of the camera (110).

[0217] Referring to FIG. 9, the camera (110) may be built into the top of the main body (90). The 90 of FIG. 9 may include the side of the main body (90) of the refrigerator (100).

[0218] Referring to 910 in FIG. 9, the refrigerator (100) can turn on the camera to start shooting when it identifies that the door is open. Also, referring to 920 in FIG. 9, the refrigerator (100) can control the motor so that the bar (91) attached to the camera can protrude as it identifies a second storage area into which an object is placed. The motor may be built into the main body (90) of the refrigerator and may be combined with the bar (91). Thus, the refrigerator (100) can adjust the degree of protrusion of the bar (91) by operating the motor.

[0219] The protrusion length of the camera (110) can be pre-set based on the location of the second storage area and stored in memory (110).

[0220] When the refrigerator (100) identifies that an object is placed in the third storage area among the multiple storage areas, it can identify the first protruding length corresponding to the third storage area as the protruding length of the camera (110).

[0221] Additionally, when the refrigerator (100) identifies that an object is placed in the fourth storage area among the plurality of storage areas, it can identify the second protruding length corresponding to the fourth storage area as the protruding length of the camera (110).

[0222] At this time, the height of the third storage area may be lower than the height of the fourth storage area. Also, the first protrusion length may be shorter than the second protrusion length.

[0223] For example, when an object is placed in a storage area corresponding to the first floor among multiple storage areas, the protrusion length of the camera (110) may be shorter compared to when the object is placed in the second floor. As the object is placed in a higher floor among multiple storage areas, the refrigerator (100) can control the bar (91) so that the camera (110) protrudes longer to photograph the storage room (900).

[0224] Based on identifying the protruding length of the camera (110), the refrigerator (100) can identify the coordinates of the object in real time using the camera (110) until the object is fully stored in the second storage area.

[0225] In the following description, the operation of the refrigerator (100) using the camera (110) to identify the coordinates of an object in real time until the object is fully stored in the second storage area is described as tracking.

[0226] The refrigerator (100) can identify the coordinates of an object using the first image. The method by which the refrigerator (100) identifies the coordinates of an object is the same as described above. Additionally, the example described above assumes a case where there is only one object being placed in the refrigerator (100).

[0227] The refrigerator (100) can identify multiple objects in the first image obtained. For example, the refrigerator (100) can identify up to six objects in the first image obtained. The refrigerator (100) can identify the object to be tracked among the identified multiple objects and perform tracking.

[0228] In one embodiment, it is assumed that two objects are received simultaneously. The refrigerator (100) can identify the object that is received into the refrigerator (100) first among the two objects as the object to be tracked.

[0229] Additionally, while the refrigerator (100) is tracking an object identified using the camera (110), if the coordinates of the object are not identified for more than a preset time, it can identify that tracking of another object will be performed.

[0230] Additionally, while the refrigerator (100) is tracking the object that enters the refrigerator (100) first among two objects, there may be cases where another object reaches the storage area before the object being tracked. In this case, the refrigerator (100) can track the object that reaches the storage area first. For example, the refrigerator (100) can compare the y-coordinates of the two objects and identify the object that has a smaller y-coordinate value as the object that reaches the storage area first.

[0231] In one embodiment, it is assumed that the user (1) simultaneously takes out an object and takes in another object. The refrigerator (100) can identify the object being taken in as an object to be tracked. The refrigerator (100) can identify the taking in and taking out of an object based on the direction of movement of the object and the edge (810) for identifying the taking in or taking out of the object. The method by which the refrigerator (100) identifies the taking in and taking out of an object was described above in operation 535.

[0232] The refrigerator (100) can control the camera (110) for tracking the object until the object is fully loaded into the second storage area. For example, the refrigerator (100) can control the length of the bar (91) and the angle of the camera (110) based on the coordinates of the object.

[0233] For example, referring to 920 in FIG. 9, the refrigerator (100) can control the length of the bar (91) or the angle of the camera (110) so that the camera (110) can photograph the storage area when an object is placed in a deep position of the second storage area.

[0234] While tracking an object, the refrigerator (100) can identify that the object has been received if the object is located within a coordinate range corresponding to a second storage area and the object's coordinates do not change for a preset amount of time. When the refrigerator (100) identifies that the object has been received, it can track other objects that have not been received. Additionally, when the refrigerator (100) identifies that there are no other objects that have not been received, it can stop tracking.

[0235] In operation 545, the refrigerator (100) can acquire a second image through the camera (110) based on the protrusion length and identify the storage location of an object stored in a second storage area. The storage location of the object may include the location where the object has been received.

[0236] When the refrigerator (100) identifies that an object has been received, it can acquire a second image. The second image may be an image containing the interior of the storage area at the time when the object has been received.

[0237] Additionally, the refrigerator (100) can identify the location where the object has been received based on the second image. For example, the refrigerator (100) can identify the location where the object has been received by identifying the coordinates of the object identified in the second image.

[0238] In operation 550, the refrigerator (100) can identify that an object is stored in the second storage area. The method of identifying that an object is stored in the second storage area is the same as described above.

[0239]

[0240] *In operation 550-Y and operation 555, the refrigerator (100) can update information regarding the type of food corresponding to the second storage area and the size of the remaining space based on the storage location of the object.

[0241] The refrigerator (100) can update information regarding the type of food corresponding to the second storage area and the size of the remaining space based on the second image.

[0242] The refrigerator (100) can update information about the type of the received object based on the second image. For example, if the received object is milk, the refrigerator (100) can store information in memory (120) that milk has been received in the second storage area.

[0243] Additionally, the refrigerator (100) can compare the second image with the last image acquired before the second image was acquired using an artificial intelligence model. Accordingly, the refrigerator (100) can identify whether there is an object added or deleted in the second storage area. If there is an object added or deleted in the second storage area, the refrigerator (100) can identify the type of object added or deleted in the storage area and store information about the addition and deletion of the object in memory (120).

[0244] For example, the refrigerator (100) can identify that the tangerine has disappeared from the storage area by comparing the second image with the last image obtained before the second image was obtained. The refrigerator (100) can delete the tangerine from the data for the storage area by updating the data stored in the memory (120) that the tangerine exists in the storage area.

[0245] In addition, in one embodiment, the refrigerator (100) can identify that one egg is missing by comparing the second image with the last image acquired before the second image was acquired. Information about food types for which quantity is important can be stored in the memory (120). For example, food for which quantity is important may include eggs.

[0246] The refrigerator (100) can update data stored in the memory (120) that eggs exist in the storage area, thereby storing information about the number of remaining eggs in the memory (120). The operation of the refrigerator (100) updating information about food types where quantity is important in the memory (120) is not limited to the example described above and can also be performed when the release of an object is detected.

[0247] The refrigerator (100) can update information about the size of the remaining space corresponding to the second storage area.

[0248] The refrigerator (100) can identify a first size of an object being received based on a first image. The first size may include the lengths of the width, height, and depth of the object identified based on coordinate values.

[0249] For example, the refrigerator (100) can identify coordinates corresponding to each vertex of an object when an object is received. As described above, the refrigerator (100) can identify the coordinates of the object using an artificial intelligence model.

[0250] The refrigerator (100) can identify the second size of the object by applying a weight corresponding to the identified storage area to the first size.

[0251] Referring to FIG. 10a, the refrigerator (100) can identify the width (1100), height (1200), and depth (1300) of an object based on the vertices of the object (1000). Since the refrigerator (100) has a camera (110) that takes pictures from top to bottom, the closer the object (1000) is to the camera (110), the larger the object (1000) appears in the picture compared to its actual size, and the further the object is from the camera (110), the smaller the object (1000) appears in the picture compared to its actual size. Therefore, the refrigerator (100) can apply weights to the values ​​of the width (1100), height (1200), and depth (1300) of the object (1000) identified based on the coordinates.

[0252] To identify the weight, the refrigerator (100) can identify the storage area where the object (1000) is received among the multiple storage areas of the refrigerator (100).

[0253] The refrigerator (100) can identify a weight based on the storage area into which an object is placed. The refrigerator (100) may have a smaller value for the weight applied to the first size as the height of the storage area increases. For example, the refrigerator (100) can identify a weight with a smaller value compared to the storage area corresponding to the second floor when an object (1000) is placed in the storage area corresponding to the first floor.

[0254] When the refrigerator (100) identifies a weight, it can apply the identified weight to the width, height, and depth of the object (1000) identified based on coordinates. The refrigerator (100) can identify a second size of the object (1000) by applying the weight to a first size. The second size may include a size that the refrigerator (100) identifies as the actual size of the object (1000).

[0255] There may be cases where the refrigerator (100) identifies only a portion of the width, height, and depth of an object (1000) based on a first image. For example, there may be cases where the object (1000) is received in a state where it overlaps with another object. In this case, the first image may be acquired with a portion of the object (1000) obscured. In this case, the refrigerator (100) may identify only a portion of the width, height, and depth of the object.

[0256] If the refrigerator (100) has unidentified lengths among the width, length, and height, it can identify the length of the object based on the second image.

[0257] For example, if the refrigerator (100) identifies that the size of the object (1000) is not identified based on the first image, it can identify the size of the object based on the similarity ratio between the object (1000) and the storage area where the object is stored.

[0258] Referring to FIG. 10b, the refrigerator (100) can acquire a second image including the state in which an object (1000) has been received. Information regarding the width, length, and height between each shelf of the shelf included in each storage area of ​​the refrigerator can be stored in memory (120).

[0259] The refrigerator (100) can identify the unidentified length among the sizes of the object by considering the ratio of the width, length, and height of the shelf as the same as the ratio of the width, length, and height of the object.

[0260] There may be cases where the refrigerator (100) cannot determine the size of an object based on the methods described above. In this case, the refrigerator (100) can identify the identified size among the object's sizes as width, height, and depth. For example, if the refrigerator (100) identifies only the width, it can identify the height and depth as being the same length as the width. Additionally, if the refrigerator (100) identifies only the lengths of two edges among the object's lengths, it can identify the longer length among the two edges as the length of the unidentified edge.

[0261] The refrigerator (100) can identify the volume of an object based on the width, length, and height of the object.

[0262] The refrigerator (100) can update information about the size of the remaining space based on identifying the size of the object.

[0263] The refrigerator (100) can identify the size of the remaining space corresponding to the second storage area based on the size of the incoming object.

[0264] The refrigerator (100) can identify the area of ​​a second storage area stored in memory (120). The refrigerator (100) can identify the remaining area of ​​the second storage area based on the difference between the area of ​​the identified object and the area of ​​the second storage area. The area of ​​the identified object may include the value obtained by multiplying the width and height. Additionally, the remaining area may include the area of ​​the second storage area where food is not located.

[0265] If the refrigerator (100) has information about the remaining area corresponding to the second storage area stored in the memory (120), it can identify the difference in the area of ​​the object identified from the stored remaining area.

[0266] The refrigerator (100) can identify the identified value as the area of ​​the second storage area.

[0267] The refrigerator (100) can identify the remaining height of the second storage area by identifying the remaining area of ​​the second storage area. The remaining height may include the height of the area in the second storage area where food is not located.

[0268] The refrigerator (100) can identify the vertical length of food identified as being stored in the second storage area in memory (120). The refrigerator (100) can identify the remaining height based on the difference between the height of the second storage area and the vertical length of food identified as being stored in the second storage area.

[0269] If the refrigerator (100) cannot identify the vertical length of food identified as being loaded in the storage area, the vertical length of the food can be identified as half the height of the second storage area. For example, if the refrigerator (100) cannot identify the vertical length of the food when the height of the second storage area is 1m, the vertical length of the food can be identified as 50cm.

[0270] The refrigerator (100) can store information about the identified remaining area and remaining height in memory (120). Additionally, the refrigerator (100) can display the type of food contained in each storage area on the display (171) in memory (120).

[0271] For example, referring to 1110 in FIG. 11a, the refrigerator (100) can perform labeling (e.g., ①, ②, ③, ④, ⑤, ⑥, ⑦, ⑧) on each of the multiple storage areas.

[0272] Referring to 1120 in FIG. 11a, the refrigerator (100) can display a user interface on the display (171) indicating the type of food contained in each storage area based on the labeling of the storage area. At this time, the storage area corresponding to the label of the storage area may correspond to the label displayed on the user interface. For example, the storage area corresponding to ① in 1110 of FIG. 11a may represent ① in 1120 of FIG. 11a.

[0273] Additionally, the refrigerator (100) may display icons corresponding to the types of food stored in each storage area on the user interface. For example, the refrigerator (100) may display an egg icon at ① of 1120 in FIG. 11a. This may include information that an egg is stored in the storage area corresponding to ① of 1110.

[0274] The refrigerator (100) can update the user interface based on the information about the object and storage area in the memory (120).

[0275] For example, when an object is placed in each storage area of ​​the refrigerator (100), an icon corresponding to the placed object can be displayed in the user interface in the area corresponding to the storage area where the object was placed.

[0276] Additionally, when an object is released from each storage area, the refrigerator (100) may stop displaying the icon corresponding to the released object among the icons in the area corresponding to the storage area from which the object was released in the user interface.

[0277] The user (1) can input user input for the icon and move and change the icon.

[0278] For example, when a touch input for an icon is received, the refrigerator (100) can move the icon based on the touch input.

[0279] Additionally, the refrigerator (100) may display a button to delete an icon on the display (171) when a touch input to the icon is received for a preset time or longer. The refrigerator (100) may stop displaying the icon on the display (171) when a touch input to the button to delete an icon is received.

[0280] Additionally, there may be cases where the refrigerator (100) fails to identify at least one of the object type and the storage area where the object was received. The refrigerator (100) may store objects for which the storage area was not identified in an unidentified list. Additionally, the refrigerator (100) may display the unidentified list on the display (171).

[0281] When a touch input for an unidentified list is received, the refrigerator (100) can display a list of objects included in the unidentified list on the display (171). The objects included in the unidentified list may include objects that have not been identified in the storage area received as described above.

[0282] Objects included in the unidentified list may be displayed with icons corresponding to the type of each object. For example, referring to FIG. 11b, the refrigerator (100) may identify that the multiple objects received are lemon, pork, and beef, but may not identify the storage area in which each object was received. The refrigerator (100) may include lemon (1122), pork (1123), and beef (1124) in the unidentified list (1121). Additionally, when the refrigerator (100) receives a touch input for the unidentified list (1121), it may display icons corresponding to each of the lemon (1122), pork (1123), and beef (1124) on the display (171).

[0283] When a touch input is received for moving an icon to one of the storage areas, the refrigerator (100) can move the icon to one of the storage areas.

[0284] Touch input for moving an icon to one of a plurality of storage areas may include the action of the user selecting an icon and dragging it to the storage area to which it is to be moved. For example, referring to FIG. 11b, the refrigerator (100) can move a lemon icon (1122) to an area corresponding to storage area ③ among the interface screens corresponding to a plurality of storage areas based on user input.

[0285] By moving the position of the icon of an identified object based on user input, the refrigerator (100) can move the icon of an object that the refrigerator has incorrectly stored or has not recognized as a storage location to the location of the actual object stored in the storage room. Accordingly, the user can manage the types of objects stored in the storage room more conveniently.

[0286] In operation 550-N, 560, the refrigerator (100) may provide a second notification to guide the first storage area.

[0287] If the refrigerator (100) is not identified as having an object stored in the second storage area, it may provide a second notification to guide to the first storage area.

[0288] For example, the refrigerator (100) may identify an incoming object and provide a notification regarding the optimal storage area, but the user (1) may not store the object in the optimal storage area. If the refrigerator (100) identifies that the user (1) is not storing the object in the optimal storage area, it may provide a notification again instructing the user to store the object in the optimal storage area. As described above, the notification may include voice notifications, cabinet pop-up notifications, lighting notifications, and display notifications.

[0289] For example, if the object is a vegetable, the refrigerator (100) may guide the user (1) to store the object in a storage area that operates in a vegetable-only mode while the object is being received. If the refrigerator (100) has provided guidance to the user (1) but the object is not stored in a storage area that operates in a vegetable-only mode, it may provide a notification again guiding the user to store the object in a vegetable-only mode.

[0290] If the refrigerator (100) does not move the object even after providing a notification, it can acquire a second image of the object and update the database for the type of the object and the remaining storage area of ​​the storage area where the object is stored.

[0291] After the refrigerator (100) updates the database, if it detects that the operation has ended, it can move the protruding camera (110) into the main body.

[0292] The refrigerator (100) can identify that the operation has ended when it identifies that the door is closing.

[0293] For example, the refrigerator (100) can obtain an image of the door using a camera (110). Based on the image, the refrigerator (100) can identify lines located at the top of the main body and the top of the door.

[0294] The refrigerator (100) can identify the angle of the door and the change in the length of the line included at the top of the door in the image. The refrigerator (100) can identify whether the door is closing by inputting the angle of the door and the change in the length of the line included at the top of the door into an artificial intelligence model.

[0295] In addition, in one embodiment, the refrigerator (100) can identify that the door of the refrigerator (100) is closing using an angle sensor. The angle sensor can measure a change in the rotation axis contained in the door while the door is closing. The angle sensor can detect an angle based on the change in the rotation axis.

[0296] The refrigerator (100) can use an angle sensor to identify that the door is closed when the angle detected by the sensor is less than a preset angle.

[0297] When the refrigerator (100) identifies that the door is closing, the bar (91) can be adjusted to move the camera (110) into the main body.

[0298] Although various embodiments have been described above, each embodiment is not necessarily implemented individually, and may be combined with at least one other embodiment, either wholly or partially, to be implemented together in a single product.

[0299] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include other data of modulated data signals, such as computer-readable instructions, data structures, or program modules.

[0300] Additionally, computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage media' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, 'non-transitory storage media' may include a buffer in which data is stored temporarily.

[0301] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0302] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0303] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.

Claims

1. Regarding refrigerators, camera; Memory for storing instructions; and at least one processor including processing circuitry; and When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, When the door of the refrigerator is opened, a first image is obtained through the camera, and Identify the type and size of an object being placed in the refrigerator based on the first image above, and Based on the type of food stored in each of the plurality of storage areas of the refrigerator and the size of the remaining space, and the type and size of the object, a first storage area corresponding to the object is identified among the plurality of storage areas, and A refrigerator that provides a first notification to guide the above-mentioned first storage area.

2. In Paragraph 1, The camera is movably installed on the refrigerator so as to protrude outward from the refrigerator, and When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, Based on the first image above, identify the second storage area where the object is received among the plurality of storage areas, and Identify the protrusion length of the camera based on the location of the second storage area, and A second image is obtained through the camera protruding based on the above protrusion length, and Identifying the storage location of the object stored in the second storage area based on the second image above, and A refrigerator that updates information regarding the type of food and the size of the remaining space corresponding to the second storage area stored in the memory based on the storage location of the above object.

3. In Paragraph 2, When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, A refrigerator that provides a second notification to guide the first storage area when the second storage area is identified as being different from the first storage area.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, If it is identified that the above object is received in the third storage area among the plurality of storage areas, the first protrusion length corresponding to the third storage area is identified as the protrusion length of the camera, and If the object is identified as being received in the fourth storage area among the plurality of storage areas, the second protrusion length corresponding to the fourth storage area is identified as the protrusion length of the camera, and In the refrigerator above, the height of the third storage area is lower than the height of the fourth storage area, and A refrigerator in which the first protrusion length is shorter than the second protrusion length.

5. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, Detecting multiple edges corresponding to multiple storage areas in an image acquired through the above camera, and A refrigerator that identifies a storage area among a plurality of storage areas where an object is placed in the refrigerator based on the plurality of edges.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, Identify whether the type of the above object corresponds to a specific food type, and A refrigerator that determines a dedicated storage area for storing the specific food type as the first storage area based on being identified as corresponding to the specific food type.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, A refrigerator that identifies the first storage area among the plurality of storage areas based on the type of object stored in each of the plurality of storage areas and the type of the object.

8. In Paragraph 7, When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, Based on the type of object stored in each of the plurality of storage areas and the type of the object, if it is identified that among the plurality of storage areas, there is no storage area where an object of the same type as the object is stored, then among the plurality of storage areas, a storage area where the object has a history of being stored is identified. A refrigerator that identifies the first storage area based on the size of the object and the size of the remaining space of the storage area where the object has a history of being stored.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, Identifying the first size of the object based on the first image above, and Identifying the storage area where the object is placed among the plurality of storage areas of the refrigerator, and A refrigerator that identifies a second size of an object by applying a weight corresponding to the identified storage area to a first size.

10. In Paragraph 9, When the above instructions are executed individually or collectively by the at least one processor, the refrigerator, A refrigerator that identifies the second size based on the similarity ratio between the object and the storage area in which the object is stored, based on the fact that the first size is not identified based on the first image.

11. In a method for controlling a refrigerator, When the door of the refrigerator is opened, a step of acquiring a first image through a camera; A step of identifying the type and size of an object being placed in the refrigerator based on the first image above; A step of identifying a first storage area corresponding to an object among the plurality of storage areas based on the type of food stored in each of the plurality of storage areas of the refrigerator and the size of the remaining space, and the type and size of the object; and A control method comprising the step of providing a first notification for guiding the first storage area.

12. In Paragraph 11, The camera is movably installed on the refrigerator so as to protrude outward from the refrigerator, and The above control method is, A step of identifying a second storage area in which the object is stored among the plurality of storage areas based on the first image; A step of identifying the protrusion length of the camera based on the location of the second storage area; A step of acquiring a second image through the camera protruding based on the protrusion length; A step of identifying the storage location of the object stored in the second storage area based on the second image; and A control method further comprising the step of updating information regarding the type of food and the size of the remaining space corresponding to the second storage area based on the storage location of the object.

13. In Paragraph 12, A control method further comprising the step of providing a second notification to guide the first storage area when the second storage area is identified as being different from the first storage area.

14. In Paragraph 11, The step of identifying the protrusion length of the above camera is, If the object is identified as being received in a third storage area among the plurality of storage areas, a step of identifying a first protrusion length corresponding to the third storage area as the protrusion length of the camera; and If the object is identified as being received in the fourth storage area among the plurality of storage areas, the method includes the step of identifying the second protrusion length corresponding to the fourth storage area as the protrusion length of the camera; A control method in which the height of the third storage area in the above refrigerator is lower than the height of the fourth storage area.

15. In Paragraph 11, The step of identifying the second storage area where the above object is received is, A step of detecting a plurality of edges corresponding to a plurality of storage areas in an image acquired through the above camera; and A control method comprising the step of identifying, among the plurality of storage areas, a storage area where the object is placed in the refrigerator based on the plurality of edges.

Images