Method for providing image of refrigerator door and refrigerator therefor

Abstract

Disclosed is a method comprising the steps of: acquiring a front view image of a refrigerator door by transforming a viewpoint corresponding to the pose of a camera positioned inside a main body into the viewpoint at which the camera faces the refrigerator door from the front of the refrigerator; separating regions of a plurality of door bins included in the front view image of the refrigerator door; obtaining a scale adjustment ratio corresponding to each of the regions of the plurality of door bins on the basis of the pose of the camera; and displaying the front view image of the refrigerator door in which the respective scales of the regions of the plurality of door bins have been adjusted on the basis of the scale adjustment ratios.

Description

Method for providing refrigerator door video and a refrigerator for this purpose

[0001] The present disclosure relates to a method for correcting and providing a refrigerator door image captured by an internal camera.

[0002] A refrigerator is a device designed to store food for a long time without spoilage, using freezing or refrigeration methods. Fundamentally, a refrigerator functions to maintain the internal temperature at a set level by creating a cooling environment using a refrigerant.

[0003] Recently, as large amounts of food are stored in refrigerator storage spaces, there is a demand for technologies that allow users to easily manage the stored food.

[0004] A method for a refrigerator to provide a refrigerator door image according to one embodiment of the present disclosure may include: a step of detecting a plurality of lines in a refrigerator door image including a plurality of door bins obtained using a camera located inside the main body of the refrigerator; a step of obtaining a camera pose using the detected plurality of lines; a step of obtaining a front image of the refrigerator door by converting a viewpoint corresponding to the camera pose to a viewpoint in which the camera looks directly at the refrigerator door; a step of separating the areas of the plurality of door bins included in the front image of the refrigerator door; a step of obtaining a scale adjustment ratio corresponding to each of the areas of the plurality of door bins based on the camera pose; and a step of displaying a front image of the refrigerator door in which the scale of each of the areas of the plurality of door bins is adjusted based on the scale adjustment ratio.

[0005] A refrigerator according to one embodiment of the present disclosure may include: a camera located inside a main body; a memory storing one or more instructions; and at least one processor operably coupled to the memory and including a processing circuit. By executing one or more instructions, either alone or in cooperation with at least one processor, the refrigerator may detect a plurality of lines in a refrigerator door image including a plurality of door bins acquired using the camera. By executing one or more instructions, either alone or in cooperation with at least one processor, the refrigerator may acquire a pose of the camera using the detected plurality of lines. By executing one or more instructions, either alone or in cooperation with at least one processor, the refrigerator may acquire a frontal image of the refrigerator door by converting a viewpoint corresponding to the pose of the camera to a viewpoint where the camera faces the refrigerator door directly. By executing one or more instructions, either alone or in cooperation with at least one processor, the refrigerator may separate a region of a plurality of door bins included in the frontal image of the refrigerator door. By having at least one processor execute one or more instructions, either alone or in cooperation, the refrigerator can acquire a scaling ratio corresponding to each of the zones of the plurality of door bins based on the pose of the camera. By having at least one processor execute one or more instructions, either alone or in cooperation, the refrigerator can display a front image of the refrigerator door scaled to each of the zones of the plurality of door bins based on the scaling ratio.

[0006] FIG. 1 is a drawing for explaining a refrigerator according to one embodiment of the present disclosure.

[0007] FIG. 2 is a flowchart illustrating a method for a refrigerator to provide a refrigerator door image according to one embodiment of the present disclosure.

[0008] FIG. 3 is a diagram illustrating the operation of a refrigerator according to one embodiment of the present disclosure estimating the open angle of a refrigerator door using a histogram.

[0009] FIG. 4 is a drawing illustrating the operation of a refrigerator according to one embodiment of the present disclosure estimating the open angle of a refrigerator door using a vanishing point.

[0010] FIG. 5 is a drawing for illustrating a refrigerator door image in which distortion correction has been performed according to one embodiment of the present disclosure.

[0011] FIG. 6 is a drawing for explaining the operation of a refrigerator according to one embodiment of the present disclosure in determining a horizontal vanishing point.

[0012] FIG. 7 is a diagram illustrating the operation of a refrigerator according to one embodiment of the present disclosure detecting a plurality of lines within defined regions of interest.

[0013] FIG. 8 is a diagram illustrating the operation of a refrigerator determining a rotation matrix according to one embodiment of the present disclosure.

[0014] FIG. 9 is a diagram illustrating the operation of cropping a refrigerator door area in a three-dimensionally rotated refrigerator door image of a refrigerator according to one embodiment of the present disclosure.

[0015] FIG. 10 is a drawing for illustrating a front view of a refrigerator door according to one embodiment of the present disclosure.

[0016] FIG. 11 is a drawing for explaining the operation of obtaining a scale adjustment ratio corresponding to each of the zones of a plurality of door bins according to one embodiment of the present disclosure.

[0017] FIG. 12 is a drawing for illustrating a front view of a refrigerator door with the scale adjusted for each of the zones of a plurality of door bins according to one embodiment of the present disclosure.

[0018] FIG. 13 is a diagram illustrating the operation of a refrigerator according to one embodiment of the present disclosure acquiring a refrigerator door image at a different angle based on a front image of the refrigerator door.

[0019] FIG. 14 is a flowchart for explaining a method for displaying a front view of a refrigerator door including a background image corresponding to each of the zones of a plurality of door bins according to one embodiment of the present disclosure.

[0020] FIG. 15 is a drawing for explaining the operation of a refrigerator according to one embodiment of the present disclosure determining the size of a mask area.

[0021] FIG. 16 is a diagram illustrating the operation of applying mask area information and images of each of the areas of a plurality of doorbins to a generative AI model according to one embodiment of the present disclosure.

[0022] FIG. 17 is a drawing for illustrating a front view of a refrigerator door including a background image corresponding to each of the zones of a plurality of door bins according to one embodiment of the present disclosure.

[0023] FIG. 18 is a drawing for explaining the operation of generating a background image including a shadow in a mask area according to one embodiment of the present disclosure.

[0024] FIG. 19 is a block diagram illustrating the function of a refrigerator according to one embodiment of the present disclosure.

[0025] FIG. 20 is a detailed block diagram for explaining the function of a refrigerator according to one embodiment of the present disclosure.

[0026] FIG. 21 is a drawing for explaining a communication system of a refrigerator according to one embodiment of the present disclosure.

[0027] The terms used in this disclosure will be briefly explained, and an embodiment of this disclosure will be described in detail.

[0028] The terms used in this disclosure have been selected to be as widely used as possible, taking into account the functions in the embodiments of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description section of the relevant embodiments of this disclosure. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0029] In the present disclosure, the expression “at least one of a, b, or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b, and c all”, or variations thereof.

[0030] Throughout the entire disclosure, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "module," etc., as described in the disclosure refer to a unit that processes at least one function or operation, and "...part" or "module" may be implemented in hardware or software, or a combination of hardware and software.

[0031] It should be understood that the blocks in each flowchart and combinations of flowcharts can be executed by one or more computer programs containing computer-executable instructions. One or more computer programs may be stored all in a single memory or may be partitioned and stored in multiple different memories.

[0032] Unless the context clearly indicates otherwise, the singular forms (e.g., "a," "an," and "the") may be understood to include plural objects. Thus, for example, the description "a component surface" may include cases where it refers to one or more of such surfaces.

[0033] All functions or operations described in this document may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuitry that performs processing and may include circuitry such as an AP (Application Processor), CP (Communication Processor), GPU (Graphical Processing Unit), NPU (Neural Processing Unit), MPU (Microprocessor Unit), SoC (System on Chip), IC (Integrated Chip), etc.

[0034] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, an embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiment described herein. Furthermore, in order to clearly explain an embodiment of the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the present disclosure are denoted by similar reference numerals.

[0035] A refrigerator according to one embodiment may include a main body.

[0036] The "main body" may include an inner body, an outer body positioned on the outside of the inner body, and an insulating material provided between the inner body and the outer body.

[0037] The "inner body" may include at least one of a case, plate, panel, or liner forming a storage chamber. The inner body may be formed as a single body or may be formed by assembling multiple plates. The "outer body" may form the exterior of the main body and may be coupled to the outer side of the inner body so that an insulating material is disposed between the inner body and the outer body.

[0038] The "insulating material" can insulate the interior and exterior of the storage room so that the temperature inside the storage room is maintained at a set appropriate temperature without being affected by the external environment. According to one embodiment, the insulating material may include a foamed insulating material. The foamed insulating material can be formed by injecting and foaming urethane foam, which is a mixture of polyurethane and a foaming agent, between the inner and outer layers.

[0039] According to one embodiment, the insulation material may additionally include a vacuum insulation material in addition to a foam insulation material, or the insulation material may consist solely of a vacuum insulation material instead of a foam insulation material. The vacuum insulation material may include a core material and an outer shell material that accommodates the core material and seals the interior under vacuum or near-vacuum pressure. However, the insulation material is not limited to the foam insulation material or vacuum insulation material described above and may include various materials that can be used for insulation.

[0040] The "storage room" may include a space defined by an internal structure. The storage room may further include an internal structure defining a space corresponding to the storage room. Various items such as food, medicine, and cosmetics may be stored in the storage room, and the storage room may be formed so that at least one side is open to allow for the retrieval and retrieval of items.

[0041] A refrigerator may include one or more storage compartments. When two or more storage compartments are formed in a refrigerator, each storage compartment may have a different use and may be maintained at a different temperature. To this end, each storage compartment may be partitioned from one another by a partition containing insulation.

[0042] The storage room may be provided to be maintained within an appropriate temperature range according to its intended use and may include a "refrigeration room," "freezing room," or "variable temperature room" distinguished according to its intended use and / or temperature range. The refrigerator room may be maintained at a temperature suitable for refrigerated storage of goods, and the freezer room may be maintained at a temperature suitable for frozen storage of goods. "Refrigeration" may mean cooling goods to a temperature that does not freeze them; for example, the refrigerator room may be maintained within a range of 0 degrees Celsius to 7 degrees Celsius. "Freezing" may mean cooling goods to freeze them or to maintain them in a frozen state; for example, the freezer room may be maintained within a range of -20 degrees Celsius to -1 degree Celsius. The variable temperature room may be used as either a refrigerator room or a freezer room, with or without the user's choice.

[0043] Storage rooms may be referred to by various names, such as "vegetable room," "fresh room," "cooling room," and "ice-making room," in addition to terms like "refrigeration room," "freezing room," and "variable temperature room." The terms "refrigeration room," "freezing room," and "variable temperature room" used below should be understood as encompassing storage rooms with corresponding uses and temperature ranges.

[0044] According to one embodiment, the refrigerator may include at least one door configured to open and close one side of the storage compartment. The door may be provided to open and close each of one or more storage compartments, or a single door may be provided to open and close multiple storage compartments. The door may be installed to be rotatable or sliding on the front of the main body.

[0045] The “door” may be configured to seal the storage room when the door is closed. The door may include insulation material, similar to the main body, to insulate the storage room when the door is closed.

[0046] According to one embodiment, the door may include a door outer panel forming the front of the door, a door inner panel forming the rear of the door and facing the storage room, an upper cap, a lower cap, and a door insulation material provided inside the same.

[0047] A gasket may be provided on the edge of the door inner panel to seal the storage compartment by adhering to the front of the main body when the door is closed. The door inner panel may include a dyke that protrudes rearward to allow a door basket for storing items to be mounted.

[0048] According to one embodiment, the door may include a door body and a front panel detachably coupled to the front side of the door body and forming the front of the door. The door body may include a door outer panel forming the front of the door body, a door inner panel forming the rear of the door body and facing the storage compartment, an upper cap, a lower cap, and a door insulation material provided inside them.

[0049] Refrigerators can be classified into French Door Type, Side-by-side Type, BMF (Bottom Mounted Freezer), TMF (Top Mounted Freezer), or 1-door refrigerators depending on the arrangement of the door and storage compartment.

[0050] According to one embodiment, the refrigerator may include a cold air supply device arranged to supply cold air to the storage compartment.

[0051] The "cold air supply device" may include a machine, apparatus, electronic device, and / or a system combining these that can generate cold air and guide cold air to cool a storage room.

[0052] According to one embodiment, a cold air supply device can generate cold air through a refrigeration cycle that includes the compression, condensation, expansion, and evaporation processes of a refrigerant. To this end, the cold air supply device may include a refrigeration cycle device having a compressor, a condenser, an expansion device (expander), and an evaporator capable of driving the refrigeration cycle. According to one embodiment, the cold air supply device may include a semiconductor such as a thermoelectric element. The thermoelectric element can cool a storage chamber through heat generation and cooling action via the Peltier effect.

[0053] According to one embodiment, the refrigerator may include a machine room arranged to accommodate at least some parts belonging to a cold air supply device.

[0054] The "machine room" may be configured to be partitioned and insulated from the storage room to prevent heat generated from components placed in the machine room from being transferred to the storage room. The interior of the machine room may be configured to communicate with the exterior of the main body to dissipate heat from components placed inside the machine room.

[0055] According to one embodiment, the refrigerator may include a dispenser provided on the door to provide water and / or ice. The dispenser may be provided on the door so that it is accessible to a user without opening the door.

[0056] According to one embodiment, the refrigerator may include an ice-making device configured to generate ice. The ice-making device may include an ice-making tray that stores water, an ice-removing device that separates ice from the ice-making tray, and an ice bucket that stores the ice generated from the ice-making tray.

[0057] According to one embodiment, the refrigerator may include a control unit for controlling the refrigerator.

[0058] The "control unit" may include a memory that stores or remembers a program and / or data for controlling a refrigerator, and a processor that outputs a control signal for controlling a cold air supply device, etc., according to the program and / or data stored in the memory.

[0059] The memory stores or records various information, data, commands, programs, etc., necessary for the operation of the refrigerator. The memory can store temporary data generated while generating control signals to control the components included in the refrigerator. The memory may include at least one of volatile memory or non-volatile memory, or a combination thereof.

[0060] The processor controls the overall operation of the refrigerator. The processor can control the components of the refrigerator by executing programs stored in memory. The processor may include a separate NPU that performs the operation of an artificial intelligence model. Additionally, the processor may include a central processing unit, a graphics processing unit (GPU), etc. The processor can generate control signals to control the operation of the cold air supply unit. For example, the processor can receive temperature information of the storage compartment from a temperature sensor and generate a cooling control signal to control the operation of the cold air supply unit based on the temperature information of the storage compartment.

[0061] Additionally, the processor can process user input of the user interface and control the operation of the user interface according to programs and / or data stored in memory. The user interface may be provided using an input interface and an output interface. The processor can receive user input from the user interface. Additionally, the processor can transmit display control signals and image data to the user interface to display an image on the user interface in response to the user input.

[0062] The processor and memory may be provided as a single unit or separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one sub-processor. The memory may include one or more memory units.

[0063] According to one embodiment, the refrigerator may include a processor and memory that control all components included in the refrigerator, and may include a plurality of processors and a plurality of memories that individually control the components of the refrigerator. For example, the refrigerator may include a processor and memory that control the operation of a cold air supply device according to the output of a temperature sensor. Additionally, the refrigerator may separately provide a processor and memory that control the operation of a user interface according to user input.

[0064] The communication module (communication interface) can communicate with external devices, such as servers, mobile devices, and other home appliances, through nearby Access Points (APs). The Access Point (AP) can connect the Local Area Network (LAN) to which the refrigerator or user device is connected to the Wide Area Network (WAN) to which the server is connected. The refrigerator or user device can be connected to the server through the Wide Area Network (WAN).

[0065] The input interface may include keys, touchscreens, microphones, etc. The input interface may receive user input and transmit it to the processor.

[0066] The output interface may include a display, a speaker, etc. The output interface can output various notifications, messages, information, etc. generated by the processor.

[0067] Refrigerators according to various embodiments will be described in detail below with reference to the attached drawings.

[0068] FIG. 1 is a drawing for explaining a refrigerator (1000) according to one embodiment of the present disclosure.

[0069] According to one embodiment of the present disclosure, a refrigerator (1000) may be an electronic device (or home appliance) for refrigerating or freezing food. The refrigerator (1000) may store not only food but also medicine, alcoholic liquor, or cosmetics.

[0070] Referring to FIG. 1, a refrigerator (1000) according to one embodiment of the present disclosure may include at least one camera. For example, the refrigerator (1000) may include a camera (1400) at the top center of the main body. In the present disclosure, the case where the camera (1400) is an RGB camera is described as an example, but is not limited thereto. For example, the camera (1400) may be a thermal imaging camera, a depth camera, or a DVS (Dynamic Vision Sensor) camera.

[0071] The refrigerator (1000) can monitor objects entering the refrigerator (1000) or objects exiting the refrigerator (1000) through a camera (1400). In the present disclosure, objects are elements that can be stored in the refrigerator (1000), and may include, for example, food (e.g., fresh food such as fruit, vegetables, etc., cooked food, processed products, seasonings, retort food), containers for holding side dishes or food (e.g., translucent containers, transparent containers, opaque containers), beverages (e.g., bottled water, carbonated water, yogurt, coffee, milk, etc.), canned goods, sauces (e.g., ketchup, mayonnaise, salad dressing, steak sauce, etc.), medicines, alcoholic beverages, cosmetics, etc., but are not limited thereto.

[0072] The refrigerator (1000) can recognize objects entering the refrigerator (1000), objects exiting the refrigerator (1000), a user's hand entering the refrigerator (1000), and objects selected by the user by analyzing image frames acquired by the camera (1400). For example, when the refrigerator (1000) detects the opening of the refrigerator door (10), it can activate the camera (1400) included in the refrigerator (1000) and capture image frames. The refrigerator (1000) can determine that the first object has been entered if, as a result of analyzing the image frames, the user's hand holding the first object enters the refrigerator (1000) and then only the hand exits. On the other hand, if, as a result of analyzing the image frames, the hand enters the refrigerator (1000) and then the hand exits together with the second object, the refrigerator can determine that the second object has been exited.

[0073] Referring to 101 in FIG. 1, the refrigerator (1000) may provide a refrigerator storage list (11) representing objects received in the refrigerator (1000). For example, when the refrigerator (1000) detects an object received in the refrigerator (1000), it may add a thumbnail image of the object to the refrigerator storage list (11). For example, the refrigerator (1000) may create a thumbnail image by cropping an object area from an image frame captured through a camera (1400) and add the thumbnail image to the refrigerator storage list (11). On the other hand, when the refrigerator (1000) detects an object being released, it may delete the thumbnail image of the released object from the refrigerator storage list (11).

[0074] Meanwhile, when the camera (1400) is positioned at the center of the front of the top table, the shooting area of ​​the camera (1400) may include not only the storage room (refrigerator room) but also the door bin located inside the refrigerator door (10). However, when the camera (1400) is positioned at the center of the front of the top table, it may be difficult for the refrigerator (1000) to accurately detect objects entering or leaving the door bin by analyzing the door bin image. Therefore, the refrigerator (1000) may directly provide the user with an image of the inside of the refrigerator door (10) including the door bin, so that the user can identify objects stored in the door bin.

[0075] Referring to 102 in FIG. 1, when the refrigerator (1000) detects that the refrigerator door (10) is open, it can activate a camera (1400) to capture an image of the inside of the refrigerator door (10) (hereinafter referred to as the refrigerator door image) (12) and provide the refrigerator door image (12) to the user. If the refrigerator (1000) includes a left door and a right door, the refrigerator (1000) can capture the left door image and the right door image respectively and provide them to the user. For example, if the user selects the left door bin (1) in the execution window of an application that provides the refrigerator interior image, the refrigerator (1000) can provide the image of the inside of the left door, and if the user selects the right door bin (2), the refrigerator (1000) can provide the image of the inside of the right door.

[0076] Meanwhile, since the refrigerator door image (12) is captured by a camera (1400) located at the top center of the main body, it may be provided in a distorted form. For example, the refrigerator door image (12) may have perspective distortion in addition to optical distortion (e.g., barrel distortion) caused by the camera (1400). In particular, greater distortion may occur as it extends toward the outer edge of the refrigerator door image (12). In this case, visibility of objects stored in the door bin may be reduced. Accordingly, according to one embodiment of the present disclosure, the refrigerator (1000) can adjust the refrigerator door image (12) to overcome the positional constraints of the camera (1400) and provide a refrigerator door image (12) with high visibility. For example, the refrigerator (1000) can improve the visibility of the refrigerator door image (12) by estimating a vanishing point in the refrigerator door image (12) and adjusting the refrigerator door image (12) using the vanishing point so that the inside of the refrigerator door (10) is viewed as far forward as possible.

[0077] Below, we will look in detail at how the refrigerator (1000) provides an optimal refrigerator door image (12) with improved visibility, with reference to FIG. 2.

[0078] FIG. 2 is a flowchart illustrating a method for a refrigerator (1000) according to one embodiment of the present disclosure to provide a refrigerator door image.

[0079] Referring to FIG. 2, a method for a refrigerator (1000) to provide a refrigerator door image may include steps S210 to S260. In one embodiment of the present disclosure, steps S210 to S260 may be executed by at least one processor included in the refrigerator (1000). The method for a refrigerator (1000) to provide a refrigerator door image is not limited to that illustrated in FIG. 2, and in one or more embodiments, additional steps not illustrated in FIG. 2 may be included, or some steps may be omitted.

[0080] In step S210, a refrigerator (1000) according to one embodiment of the present disclosure can detect a plurality of lines in a refrigerator door image including a plurality of door bins obtained using a camera (1400) located inside the main body of the refrigerator (1000).

[0081] According to one embodiment of the present disclosure, a door bin is a storage compartment on the inside of the door that is visible when the refrigerator door is opened, and may be suitable for storing frequently used items such as small bottles, condiments, milk, and beverages. A plurality of door bins may be included in the refrigerator door, and the spacing between the plurality of door bins may vary according to the user's settings.

[0082] According to one embodiment of the present disclosure, a refrigerator (1000) can acquire a refrigerator door image including a plurality of door bins by using a camera (1400) located inside the main body of the refrigerator (1000). For example, the refrigerator (1000) can detect the open or closed state of the refrigerator door through a door opening / closing detection sensor. The door opening / closing detection sensor may include a Hall sensor, a micro switch, etc., but is not limited thereto. When the refrigerator (1000) detects that the refrigerator door is opened through the door opening / closing detection sensor, it can activate the camera (1400) to acquire a refrigerator door image including a plurality of door bins. At this time, the refrigerator door image may be a left door image, a right door image, or an image including both the left door and the right door.

[0083] According to one embodiment of the present disclosure, when a refrigerator (1000) detects that the refrigerator door is opened, it can activate a camera (1400) to acquire image frames in which the inside of the refrigerator door is captured. For example, the refrigerator (1000) can acquire 30 image frames per second, but is not limited thereto. The refrigerator (1000) can analyze each image frame to estimate the open angle of the refrigerator door included in each image frame, and acquire an image frame in which the open angle of the refrigerator door is within a defined range as a refrigerator door image. The defined range may be 35° to 55°, but is not limited thereto. For example, the refrigerator (1000) can acquire an image frame in which the open angle of the refrigerator door is 45° as a refrigerator door image. This is because when the open angle of the refrigerator door is approximately 45°, the shelf is captured symmetrically, so distortion can be minimized when converted into a frontal image. That is, the refrigerator (1000) can select an image frame in which the refrigerator door is opened at an angle of 45° as a refrigerator door image to be converted into a front image.

[0084] According to one embodiment of the present disclosure, the refrigerator (1000) may estimate the open angle of the refrigerator door by utilizing the slope of a straight line component included in each image frame, or by performing vanishing point detection in each image frame. The operation of the refrigerator (1000) estimating the open angle of the refrigerator door will be examined in detail later with reference to FIGS. 3 and FIGS. 4.

[0085] Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) can acquire image frames having an open angle of the refrigerator door within a predetermined range even if the user opens and closes the refrigerator door quickly by increasing the shutter speed of the camera. For example, the refrigerator (1000) can acquire image frames without shaking when the refrigerator door is opened 35° to 55° by increasing the shutter speed of the camera (1400). The shutter speed may be, for example, 2ms, but is not limited thereto.

[0086] According to one embodiment of the present disclosure, when the camera (1400) is positioned at the top center of the main body, the refrigerator door image acquired through the camera (1400) may include the interior of the refrigerator door at an angle rather than a frontal view, and distortion may exist. In particular, it may be difficult to identify objects stored in the door bin at the bottom in the refrigerator door image. Additionally, unnecessary elements such as the user's hand, face, or kitchen floor may be included in the refrigerator door image. Therefore, in order to improve the visibility of the refrigerator door image, it is necessary to process the refrigerator door image into a frontal image.

[0087] According to one embodiment of the present disclosure, when a refrigerator (1000) obtains a refrigerator door image suitable for processing into a front image, it can perform distortion correction on the refrigerator door image. For example, the refrigerator (1000) can perform distortion correction on the refrigerator door image when the angle at which the refrigerator door is opened is within a predetermined angle range.

[0088] The refrigerator (1000) can perform camera calibration using a checkerboard image. Camera calibration refers to the process of accurately calculating the intrinsic and extrinsic parameters of the camera (1400). Through camera calibration, the relationship between the image sensor of the camera (1400) and the real world can be accurately mapped, thereby correcting distortions and errors occurring in the image. Intrinsic parameters may include parameters regarding the focal length, optical center, and radial distortion coefficient of the camera (1400) or lens system. Extrinsic parameters may include parameters indicating the relative position and direction between the camera (1400) and the object being photographed. The refrigerator (1000) can check the degree of distortion of the camera (1400) and correct the refrigerator door image by applying a distortion opposite to the distortion of the camera (1400) to the refrigerator door image.

[0089] According to one embodiment of the present disclosure, a refrigerator (1000) can detect a plurality of lines in a refrigerator door image in which distortion correction has been performed. When distortion correction is performed on the refrigerator door image, lines that are roundly distorted in the refrigerator door image can be straightened into straight lines. Accordingly, the refrigerator (1000) can accurately detect a plurality of lines in the refrigerator door image in which distortion correction has been performed. At this time, the plurality of lines may be horizontal lines with respect to the refrigerator door. According to one embodiment of the present disclosure, the refrigerator (1000) can detect a plurality of lines within defined regions of interest. The regions of interest may be areas including the border of the door bin.

[0090] In step S220, a refrigerator (1000) according to one embodiment of the present disclosure can acquire a pose of a camera (1400) using a plurality of lines detected in step S210.

[0091] According to one embodiment of the present disclosure, the refrigerator (1000) can check the pose of the camera (1400) to convert the refrigerator door image into a front image. The pose of the camera (1400) may refer to the position or orientation of the camera in 3D space. That is, the pose of the camera (1400) can define where the camera (1400) is and where the camera (1400) is looking.

[0092] According to one embodiment of the present disclosure, the refrigerator (1000) can estimate a vanishing point using a plurality of lines. For example, the refrigerator (1000) can determine a horizontal vanishing point using a plurality of lines detected in the refrigerator door image. A vanishing point may refer to a point where lines parallel to each other in the 3D world appear to converge on a 2D image plane. The vanishing point is determined by the camera pose. Therefore, by analyzing the vanishing point on the 2D image, it is possible to infer which direction the camera was looking. According to one embodiment of the present disclosure, the refrigerator (1000) can estimate the direction (Vx, Vy, Vz) of the camera (1400) from the vanishing point detected in the refrigerator door image. The direction of the camera (1400) may be the direction in which the camera (1400) is facing in three-dimensional space. In addition, since the position of the camera (1400) is fixed, the refrigerator (1000) can know the position (e.g., coordinate value) of the camera (1400). Accordingly, the refrigerator (1000) can obtain the pose of the camera (1400) when estimating the direction of the camera (1400) using a horizontal vanishing point.

[0093] Meanwhile, since the pose of the camera (1400) and the pose of the refrigerator door are relative concepts, if the pose of the camera is set to (0, 0, 0), obtaining the pose of the camera (1400) may be expressed as obtaining the pose of the refrigerator door.

[0094] In step S230, a refrigerator (1000) according to one embodiment of the present disclosure can acquire a frontal image of the refrigerator door by converting the viewpoint corresponding to the pose of the camera (1400) to a viewpoint in which the camera (1400) looks directly at the refrigerator door.

[0095] Changing the viewpoint corresponding to the pose of the camera (1400) to a viewpoint where the camera (1400) looks directly at the refrigerator door may include changing the viewpoint from an open state to a closed state.

[0096] The viewpoint may refer to a visual position or perspective in the direction or angle in which the camera (1400) is looking. According to one embodiment of the present disclosure, since the camera (1400) is located at the top center of the main body, the viewpoint corresponding to the pose of the camera (1400) may be a viewpoint looking down at the refrigerator door from above in an oblique direction.

[0097] According to one embodiment of the present disclosure, the refrigerator (1000) may determine (or restore) a rotation matrix to convert the viewpoint of the camera (1400) relative to the front of the door. Converting the viewpoint of the camera (1400) relative to the front of the door may include converting the open angle of the refrigerator door (e.g., 45°) to 0°.

[0098] The refrigerator (1000) can obtain a rotation matrix using a single vanishing point detected in the refrigerator door image. The refrigerator (1000) can restore the rotation matrix from a single vanishing point by utilizing the characteristic that the rotation axis of the refrigerator door and the camera (1400) are fixed. For example, the refrigerator (1000) can detect a horizontal vanishing point in the refrigerator door image, determine a vertical vanishing point and a depth vanishing point using the horizontal vanishing point, and determine a rotation matrix using the horizontal vanishing point, the vertical vanishing point, and the depth vanishing point. In the case of the refrigerator door, detecting a horizontal straight line is easy due to the door bean, but detecting a depth straight line is difficult due to its flat shape, and detecting a vertical straight line may also be unstable and difficult due to the characteristic that it is captured increasingly farther away as it goes down. Therefore, according to one embodiment of the present disclosure, the refrigerator (1000) can determine the rotation matrix using the horizontal vanishing point that can be detected most robustly.

[0099] According to one embodiment of the present disclosure, a refrigerator (1000) can obtain a front image of a refrigerator door by rotating the refrigerator door image in three dimensions using a rotation matrix. For example, the refrigerator (1000) can obtain a front image of a refrigerator door by converting the 2D pixels of the refrigerator door image into three dimensions (unprojection), rotating them in three dimensions using a rotation matrix, and then converting them into two dimensions (projection). That is, the refrigerator (1000) can apply an effect such as 3D rotation to the refrigerator door image by warping the 2D image (refrigerator door image) using a rotation matrix. The operation of the refrigerator (1000) obtaining a front image of the refrigerator door using a rotation matrix will be examined in detail later with reference to FIG. 8.

[0100] Since the refrigerator door image may partially include the user's face, hands, kitchen floor, etc., the front image of the refrigerator door obtained by warping the refrigerator door image may also include the user's face, hands, kitchen floor, etc. Accordingly, according to one embodiment of the present disclosure, the refrigerator (1000) can obtain a front image of the refrigerator door from which the user's face, hands, kitchen floor, etc. are removed by cropping the refrigerator door area from the front image of the refrigerator door obtained by applying a rotation matrix to the refrigerator door image. The operation of the refrigerator (1000) cropping the refrigerator door area from the front image of the refrigerator door will be examined in detail later with reference to FIG. 9.

[0101] In step S240, a refrigerator (1000) according to one embodiment of the present disclosure can separate a plurality of door bin areas included in the front image of the refrigerator door.

[0102] According to one embodiment of the present disclosure, the front image of the refrigerator door is a warped image of the refrigerator door taken from the top of the refrigerator (1000), so the front image of the refrigerator door differs from the actual front image. In particular, the difference from the actual front image becomes greater as the refrigerator door gets lower. For example, the image of a product stored in the lower door bin may appear unnaturally elongated compared to the image of a product stored in the upper door bin. Therefore, it is necessary to adjust the front image of the refrigerator door to a ratio that makes it appear as natural as possible.

[0103] According to one embodiment of the present disclosure, a refrigerator (1000) can separate areas of a plurality of door bins from a front image of the refrigerator door to adjust the front image of the refrigerator door. Since the door bins can be mounted in a plurality of slots, they can be configured in various combinations of numbers and positions to suit the user's preference. Accordingly, the refrigerator (1000) must separate areas of the actual door bins from a front image of the refrigerator door.

[0104] According to one embodiment of the present disclosure, a refrigerator (1000) can detect at least one horizontal line in a front image of the refrigerator door. For example, the refrigerator (1000) can detect a horizontal line under each door bin. The refrigerator (1000) can separate a plurality of door bin zones based on at least one horizontal line. When the refrigerator door contains three door bins, the refrigerator (1000) can distinguish a first door bin zone, a second door bin zone, and a third door bin zone in a front image of the refrigerator door.

[0105] In step S250, a refrigerator (1000) according to one embodiment of the present disclosure can obtain a scale adjustment ratio corresponding to each of the zones of a plurality of door bins based on the pose of the camera (1400).

[0106] According to one embodiment of the present disclosure, a refrigerator (1000) can determine the scale adjustment ratio of each door bin by using the position and direction of the camera (1400). The refrigerator (1000) can determine the vertical scale adjustment ratio of each of the door bin zones by the ratio between a second line perpendicular to a first line passing through the center point of each of the zones of the plurality of door bins from the camera (1400) and a third line projected from the second line with respect to the front. That is, if the angle (θ) between the second line and the third line is calculated, cosθ can be the vertical scale adjustment ratio.

[0107] Since the camera (1400) is positioned at the top center of the refrigerator body (1000), the angle between the first line and the second line can increase as it goes toward the bottom of the refrigerator door. Therefore, the refrigerator (1000) can determine the vertical scale adjustment ratio of the bottom door bin area to be larger than the vertical scale adjustment ratio of the top door bin area. The operation of the refrigerator (1000) obtaining a scale adjustment ratio corresponding to each of the multiple door bin areas will be examined in detail later with reference to FIG. 11.

[0108] In step S260, a refrigerator (1000) according to one embodiment of the present disclosure can display a front image of a refrigerator door in which the scale of each of the zones of a plurality of door bins is adjusted based on a scale adjustment ratio.

[0109] According to one embodiment of the present disclosure, the refrigerator (1000) can adjust the scale of each of the sections of a plurality of door bins based on a scale adjustment ratio. For example, if the vertical scale adjustment ratio of the first door bin section is 0.96, the refrigerator (1000) can reduce the size of the first door bin section so that the vertical length of the first door bin section becomes 96% of the total vertical length of the first door bin section. If the vertical length of the first door bin section is 340 pixels, the reduced vertical length of the first door bin section may be approximately 326 pixels. If the vertical scale adjustment ratio of the second door bin section located below the first door bin of the first door bin section is 0.73, the refrigerator (1000) can reduce the size of the second door bin section so that the vertical length of the second door bin section becomes 73% of the total vertical length of the second door bin section. If the vertical length of the second door bin area is 560 pixels, the vertical length of the reduced second door bin area may be approximately 409 pixels. If the vertical scaling ratio of the third door bin area located below the second door bin is 0.48, the refrigerator (1000) may reduce the size of the third door bin area so that the vertical length of the third door bin area becomes 48% of the total vertical length of the third door bin area. If the vertical length of the third door bin area is 850 pixels, the vertical length of the reduced third door bin area may be 442 pixels.

[0110] According to one embodiment of the present disclosure, a refrigerator (1000) can display a front view of the refrigerator door in which the scale of each of the zones of a plurality of door bins is adjusted to an appropriate ratio. According to one embodiment of the present disclosure, the refrigerator (1000) can display a front view of the refrigerator door in which the scale of each of the zones of a plurality of door bins is adjusted to an appropriate ratio on a user terminal (e.g., a smartphone) connected via a display of the refrigerator (1000) or a server (not shown). By displaying a front view of the refrigerator door in which the scale of each of the zones of a plurality of door bins is adjusted to an appropriate ratio, the refrigerator (1000) can improve the visibility of objects stored in the refrigerator door. By checking the front view of the refrigerator door displayed on the display of the refrigerator (1000) or the user terminal, the user can accurately identify the food items stored in the refrigerator door without opening the refrigerator door.

[0111] According to one embodiment of the present disclosure, a refrigerator (1000) can recognize characters in a front image of a refrigerator door containing a plurality of door bins adjusted to an appropriate ratio using an Optical Character Recognition (OCR) model. Additionally, the refrigerator (1000) can recognize objects in a front image of the refrigerator door using an object recognition model. Meanwhile, the refrigerator (1000) can detect (or track) objects entering or leaving the refrigerator door by analyzing a front image of a refrigerator door containing a plurality of door bins adjusted to an appropriate ratio. Since perspective distortion is removed from the front image of the refrigerator door, the performance of character recognition or object recognition can be improved.

[0112] Meanwhile, since the refrigerator (1000) does not have a sensor that senses the angle at which the door is opened, it can obtain an image frame that can restore the most natural frontal image as a refrigerator door image by analyzing image frames. Below, with reference to FIGS. 3 and 4, we will examine in detail the operation of the refrigerator (1000) obtaining an image frame that can restore the most natural frontal image as a refrigerator door image by estimating the angle at which the refrigerator door is opened.

[0113] FIG. 3 is a diagram illustrating the operation of a refrigerator (1000) according to one embodiment of the present disclosure estimating the angle at which the refrigerator door is opened using a histogram.

[0114] According to one embodiment of the present disclosure, when a refrigerator (1000) detects the opening of the refrigerator door, it can acquire image frames of the refrigerator door using a camera (1400) located at the top center of the main body.

[0115] Referring to 310 in FIG. 3, the refrigerator (1000) can detect straight line components in each of the image frames and calculate the angle of each straight line component. Referring to 320 in FIG. 3, the refrigerator (1000) can generate an angle histogram using the angles of each straight line component. In the angle histogram, the horizontal axis may represent the angle and the vertical axis may represent the frequency.

[0116] The refrigerator (1000) can estimate the open angle of the refrigerator door included in each image frame using an angle histogram. For example, the angle with the most linear components distributed may be the open angle of the refrigerator door. If the most linear components in the first image frame have an angle of 45°, the first image frame may be an image taken when the refrigerator door is open at 45°. If the most linear components in the second image frame have an angle of 25°, the second image frame may be an image taken when the refrigerator door is open at 25°.

[0117] According to one embodiment of the present disclosure, a refrigerator (1000) can acquire an image frame as a refrigerator door image in which the angle at which the most linear components are distributed has a value within a specific range. At this time, the specific range may be 35° to 55°, but is not limited thereto. The specific range can be defined as 35° to 55° because the shelf is captured symmetrically when the refrigerator door is open at an angle of about 45°, thereby minimizing distortion when converted into a frontal image. For example, if, as a result of analyzing the angle histogram, the most linear components in the first image frame have an angle of 45°, the refrigerator (1000) can determine the first image frame as a refrigerator door image for frontal image conversion. The refrigerator (1000) can correct the distortion of the first image frame before converting it into a frontal image.

[0118] According to one embodiment of the present disclosure, when there are multiple image frames in which the angle at which the most linear components are distributed has a value within a specific range (35° to 55°), the image frame closest to the angle at which the most linear components are distributed is 45° can be obtained as a refrigerator door image.

[0119] Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) can acquire an image frame in which the angle with the most linear components distributed is within a specific range by increasing the shutter speed of the camera (1400), even if the user opens and closes the door quickly. When the shutter speed is fast, an image frame with less blur can be acquired. For example, the refrigerator (1000) can set the shutter speed of the camera (1400) to 2ms, but is not limited thereto.

[0120] The refrigerator (1000) may not select a refrigerator door image to provide to the user and may not perform front image restoration if, among the image frames captured after detecting the opening of the refrigerator door, there is no image frame captured when the angle of opening the refrigerator door is within a specific range (e.g., 35° to 55°). For example, if the user opens the refrigerator door only up to 15° and then closes it, the refrigerator (1000) may not perform front image restoration for the image frames and may provide the user with a previously generated front image of the refrigerator door, as there is no image frame among the image frames where the angle of opening the refrigerator door is between 35° and 55°. This is because if a front image is generated using an image frame where the angle of opening the refrigerator door is 15°, there may be severe distortion in the front image, which may result in reduced visibility. Subsequently, when an image frame where the angle of opening the refrigerator door is between 35° and 55° is obtained, the refrigerator (1000) may perform front image restoration to provide a new front image of the refrigerator door.

[0121] FIG. 4 is a drawing for explaining the operation of a refrigerator (1000) according to one embodiment of the present disclosure estimating the open angle of a refrigerator door using a vanishing point.

[0122] According to one embodiment of the present disclosure, since the rotation axis of the refrigerator door and the camera (1400) are fixed, the relationship between the position of the vanishing point on the image frame obtained through the camera (1400) and the open angle of the refrigerator door can be experimentally calculated. Accordingly, when the refrigerator (1000) detects the vanishing point in the image frame, it can estimate the open angle of the refrigerator door included in the image frame using the vanishing point.

[0123] Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) can select a refrigerator door image for frontal image conversion among the image frames by utilizing the characteristic that the vanishing point of the image frames acquired when the refrigerator door is opened at a specific angle is detected within a specific range. For example, the refrigerator (1000) can set a range (400) of vanishing points detectable in the image frames when the refrigerator door is opened at a specific range (e.g., 35° to 55°). At this time, the range (400) of the vanishing points may be set in a rectangular shape, but is not limited thereto. According to one embodiment of the present disclosure, when the refrigerator (1000) detects the opening of the refrigerator door, it can acquire image frames using a camera (1400) and detect a vanishing point in each image frame. At this time, the refrigerator (1000) can select an image frame in which the vanishing point exists within a pre-set range (400) as a refrigerator door image for frontal image conversion.

[0124] According to one embodiment of the present disclosure, when there are multiple image frames in which the vanishing point exists within a preset range (400), the refrigerator (1000) can select the image frame closest to the refrigerator door opening angle of 45° based on the location of the vanishing point as the refrigerator door image.

[0125] Meanwhile, according to one embodiment of the present disclosure, the refrigerator (1000) can acquire an image frame in which the vanishing point exists within a preset range (400) even if the user opens and closes the door quickly by increasing the shutter speed of the camera (1400). When the shutter speed is fast, an image frame with less blur can be acquired. For example, the refrigerator (1000) can set the shutter speed of the camera (1400) to 2ms, but is not limited thereto.

[0126] The refrigerator (1000) may not select a refrigerator door image to provide to the user and may not perform front image restoration if, among the image frames captured after detecting the opening of the refrigerator door, there is no image frame in which the vanishing point is within a preset range (400). For example, if the user opens the left refrigerator door only up to 20° and then closes it, the refrigerator (1000) may not perform front image restoration for the image frames and may provide the previously generated front image of the refrigerator door to the user, since there is no image frame in which the vanishing point is within a preset range (400) among the image frames. This is because if a front image is generated using an image frame in which the refrigerator door is opened at an angle of 20°, there may be severe distortion in the front image, which may result in reduced visibility. Subsequently, when an image frame in which the vanishing point is within a preset range (400) is acquired, the refrigerator (1000) may perform front image restoration and provide a new front image of the refrigerator door.

[0127] FIG. 5 is a drawing for illustrating a refrigerator door image in which distortion correction has been performed according to one embodiment of the present disclosure.

[0128] According to one embodiment of the present disclosure, when a refrigerator (1000) detects the opening of a refrigerator door, it can acquire a refrigerator door image (510) using a camera (1400) and detect straight line components in the refrigerator door image (510). At this time, the refrigerator (1000) can estimate the angle at which the refrigerator door is opened based on the straight line components. For example, as described in FIGS. 3 and 4, the refrigerator (1000) can estimate the angle at which the refrigerator door is opened through an angle histogram or the estimation of the vanishing point location.

[0129] According to one embodiment of the present disclosure, the refrigerator (1000) can perform distortion correction on the refrigerator door image when the angle at which the refrigerator door is opened is within a predetermined angle range (e.g., 35° to 55°). For example, the refrigerator (1000) can perform camera calibration using a predetermined marker, such as a checkerboard. Through camera calibration, the relationship between the image sensor of the camera (1400) and the real world can be accurately mapped, thereby correcting distortion and errors occurring in the image. The refrigerator (1000) can check the degree of distortion of the camera (1400) and correct the refrigerator door image (510) by applying a distortion opposite to the distortion of the camera (1400) to the refrigerator door image (510).

[0130] Referring to FIG. 5, there are rounded distorted lines in the refrigerator door image (510). However, in the refrigerator door image (520) where distortion correction has been performed, the rounded distorted lines can be straightened into straight lines. Therefore, the refrigerator (1000) can accurately determine the vanishing point by detecting multiple lines in the refrigerator door image (520) where distortion correction has been performed. The operation of the refrigerator (1000) detecting the vanishing point in the refrigerator door image (520) where distortion correction has been performed will be examined in detail with reference to FIG. 6.

[0131] FIG. 6 is a drawing for explaining the operation of a refrigerator (1000) according to one embodiment of the present disclosure determining a horizontal vanishing point.

[0132] According to one embodiment of the present disclosure, a refrigerator (1000) can detect a plurality of lines in a refrigerator door image (520) in which distortion correction has been performed. Since the refrigerator door image (520) includes a plurality of door bins, the refrigerator (1000) can detect lines at the edges of each door bin. In the case of the refrigerator door, detection of horizontal straight lines is easy due to the door bins, but detection of depth straight lines is difficult due to the flat shape, and detection of vertical straight lines may also be unstable and difficult due to the characteristic that the image is captured increasingly farther away as it goes down. Therefore, according to one embodiment of the present disclosure, the refrigerator (1000) can determine a horizontal vanishing point using horizontal straight lines that can be detected most robustly.

[0133] Referring to FIG. 6, the refrigerator (1000) can detect a first line (601), a second line (602), and a third line (603) in a refrigerator door image (520) in which distortion correction has been performed. The refrigerator (1000) can determine the point where the first line (601), the second line (602), and the third line (603) extend and meet as a horizontal vanishing point (600). The refrigerator (1000) can estimate the direction (Vx, Vy, Vz) of the camera (1400) using the horizontal vanishing point (600).

[0134] FIG. 7 is a diagram illustrating the operation of a refrigerator (1000) according to one embodiment of the present disclosure detecting a plurality of lines within defined regions of interest.

[0135] According to one embodiment of the present disclosure, a refrigerator (1000) can detect a plurality of lines within defined regions of interest to determine a horizontal vanishing point (600). For example, the refrigerator (1000) can designate regions of interest in an image frame by utilizing the characteristic that the angle at which the refrigerator door is photographed is limited to some extent. Since the determined vanishing point may be an unstable value when the vanishing point is determined using lines detected by objects stored in the door bin, the refrigerator (1000) can set regions of interest so that lines are detected using straight components of the refrigerator door or straight components of the door bin.

[0136] For example, referring to FIG. 7, the refrigerator (1000) may designate the upper area of ​​the refrigerator door as the first area of ​​interest (710), the area including the border of the first door bin as the second area of ​​interest (720), the area including the border of the second door bin as the third area of ​​interest (730), and the area including the border of the third door bin as the fourth area of ​​interest (740). In this case, the refrigerator (1000) can perform distortion correction on the refrigerator door image as the refrigerator door image is acquired, detect a first line (701) in a first region of interest (710) of the refrigerator door image where distortion correction is performed, detect a second line (702) in a second region of interest (720) of the refrigerator door image where distortion correction is performed, detect a third line (703) in a third region of interest (730) of the refrigerator door image where distortion correction is performed, and detect a fourth line (704) in a fourth region of interest (740) of the refrigerator door image where distortion correction is performed. The refrigerator (1000) can determine the point where the first line (701), the second line (702), the third line (703), and the fourth line (704) intersect when extended as a vanishing point.

[0137] According to one embodiment of the present disclosure, the refrigerator (1000) can determine a rotation matrix for converting the viewpoint of the camera (1400) to a frontal reference using a single vanishing point. The operation of the refrigerator (1000) determining the rotation matrix will be examined in detail with reference to FIG. 8.

[0138] FIG. 8 is a diagram illustrating the operation of a refrigerator (1000) according to one embodiment of the present disclosure determining a rotation matrix.

[0139] According to one embodiment of the present disclosure, in order to obtain a frontal image of a refrigerator door, a rotation matrix calculation is required, and the rotation matrix calculation can be performed by utilizing vanishing point information. In order to reconstruct the rotation matrix using vanishing points, generally three vanishing points are required. At this time, since one vanishing point can be calculated through the cross product of two vanishing points (V3 = V1 x V2), detection of two vanishing points from the image is required.

[0140] In the case of a refrigerator door, while it is easy to detect straight lines in the horizontal direction due to the door bean, it is difficult to detect straight lines in the depth direction due to its flat shape, and it may also be unstable and difficult to detect straight lines in the vertical direction due to the characteristic that the image is captured from further away as it goes down. Therefore, according to one embodiment of the present disclosure, the refrigerator (1000) can determine the rotation matrix by using a horizontal vanishing point that can be detected most robustly. In particular, since the refrigerator door has a constant rotation axis and can operate only in a fixed environment where the camera (1400) is fixed, the rotation matrix can be determined using a single horizontal vanishing point. Therefore, since the refrigerator (1000) can restore the rotation matrix by accurately detecting only one vanishing point in the refrigerator door image, the stability of the entire algorithm can be increased.

[0141] According to one embodiment of the present disclosure, the refrigerator (1000) utilizes the feature that the rotation axis of the refrigerator door and the camera (1400) are fixed in order to restore the rotation matrix from one vanishing point.

[0142] According to one embodiment of the present disclosure, the following can be assumed to determine the rotation matrix.

[0143] 1) Referring to 810 in FIG. 8, the front of the camera can be defined as Vz, the right side of the camera as Vx, and the bottom side of the camera as Vy. The camera's orientation (Vx, Vy, Vz) can be estimated using a horizontal vanishing point (V1).

[0144] 2) The position of the camera (1400) is fixed and there is no roll. That is, it can be assumed that the camera (1400) is not tilted to the side.

[0145] 3) The refrigerator door rotates only along the hinge. In other words, there is only rotation along a single axis.

[0146] 4) The horizontal vanishing point (V1) was determined through multiple line detections as set in FIG. 6.

[0147] 5) Referring to 820 in Fig. 8, since the refrigerator door is rectangular, the horizontal vanishing point (V1) , The vertical vanishing point (V2) and the depth vanishing point (V3) are vertical. That is, V1, V2, V3 The inner product of each other is 0.

[0148] According to one embodiment of the present disclosure, a refrigerator (1000) can determine a rotation matrix by the following process using the above assumption.

[0149] 1) Referring to 830 and 840 in FIG. 8, since the refrigerator door rotates along the hinge, V2 is in the hinge direction, and since the camera (1400) is fixed without roll, V2 exists on the y, z plane of the camera (1400). That is, V2 = (0, V 2y , V 2z ) can be.

[0150] 2) Since the inner product of V1 and V2 is 0, (V 1x , V 1y , V 1z )·(0, V 2y , V 2z ) =V 2y V 1y +V 2z V 1z =0.

[0151] 3) Since V1 and V2 are direction vectors, if we ignore the scale, V 2y V 1y= -V 2z V 1z is. Therefore, V 2y =±V 1z , V 2z , V 2z = V 1y am.

[0152] 4) Since V1 is always in the front direction of the camera (Vz direction), V 1z is a positive number.

[0153] 5) V2 is Since the camera is always facing downward (Vy direction), V2 is positive.

[0154] 6) V by 3, 4, 5 2y =V 1z , V 2z =-V 1y Therefore, V2=(0, V 1z , -V 1y It becomes.

[0155] 7) The refrigerator (1000) obtains the rotation matrix after calculating V3 through V3 = V1 x V2. [V1 V2 V3] becomes the rotation matrix (R). For example, the rotation matrix (R) is This can be done. That is, if the refrigerator (1000) detects only the horizontal vanishing point (V1) in the refrigerator door image, it can obtain a rotation matrix (R) for generating a front image.

[0156] According to one embodiment of the present disclosure, since V2 is fixed in the hinge direction, a predefined value may be used. However, since the position of the camera (1400) may change slightly due to process errors occurring during the manufacturing process of the refrigerator (1000) and shaking caused by impact during actual use, in order to compensate for this, V2=(0, V 1z , -V 1y It may be more accurate to calculate it as ).

[0157] According to one embodiment of the present disclosure, a refrigerator (1000) can generate a frontal image of a refrigerator door by performing a three-dimensional rotation (e.g., Perspective Transform) as if the refrigerator door were viewed from the front. For example, the refrigerator (1000) can obtain an intrinsic matrix (K) during a camera calibration process, detect a plurality of lines in the refrigerator door image to obtain a horizontal vanishing point V1, obtain a vertical vanishing point V2 and a depth vanishing point V3 using the horizontal vanishing point V1, and obtain a rotation matrix R using V1, V2, and V3. At this time, the refrigerator (1000) can have I'=KRK -1 The front image of the refrigerator door can be obtained using I. Here, I' is the front image of the refrigerator door, and I can be the refrigerator door image. For the pixels of the refrigerator door image, K -1Multiplying by may mean unprojecting 2D screen coordinates into a point in 3D space. Multiplying a point in 3D space by K means projecting the point in 3D space into 2D screen coordinates. Therefore, the refrigerator (1000) unprojects a 2D refrigerator door image into 3D space (K -1 After multiplying by a rotation matrix (R) to rotate in 3D relative to the front, a point in the rotated 3D space is projected onto 2D screen coordinates to obtain a front image (I') of the refrigerator door.

[0158] Meanwhile, in the front image (I') of the refrigerator door obtained by rotating the refrigerator door image in three dimensions, other areas other than the refrigerator door (e.g., user's hand, floor, etc.) may exist. Therefore, the refrigerator (1000) can generate a pure front image of the refrigerator door by cropping only the refrigerator door area from the front image (I') of the refrigerator door. The operation of the refrigerator (1000) cropping only the refrigerator door area will be examined in detail with reference to FIG. 9.

[0159] FIG. 9 is a diagram illustrating the operation of cropping a refrigerator door area in a three-dimensionally rotated refrigerator door image of a refrigerator (1000) according to one embodiment of the present disclosure.

[0160] Referring to FIG. 9, since the position (x, y, z) of the camera (1400) and the size (w, h) of the refrigerator door (900) are fixed, the refrigerator (100) can know the position (x, y, z) of the camera (1400) and the size (w, h) of the refrigerator door (900) in advance. The refrigerator (1000) uses the direction vectors Vx (right side of the camera), V1 (horizontal vanishing point), and V2 (vertical vanishing point) associated with the refrigerator door (900) to back-project each pixel of the 2D refrigerator door image into a point in 3D space, and then the 3D position (P) corresponding to each vertex of the refrigerator door (900) 3d) 4 can be found. For example, if the refrigerator door image is an image frame captured when the refrigerator door (900) on the left side is opened relative to the camera (1400), the position of the top-left corner of the refrigerator door (900) is P 3d_tl = -Vx*W, and the position of the top right corner of the refrigerator door (900) is P 3d_tr = V1*W- P 3d_tl And, The position of the lower left corner of the refrigerator door (900) is P 3d_bl = V2*h- P 3d_tl and the position of the bottom right corner of the refrigerator door (900) is P 3d_br = V2*h+ P 3d_tr It could be.

[0161] The refrigerator (1000) uses a rotation matrix to obtain a three-dimensional position (P) corresponding to each vertex of the refrigerator door (900). 3d By rotating it, the refrigerator door can be moved to the vertex when viewed from the front. That is, the refrigerator (1000) is in a three-dimensional position (P 3d The rotated 3D position (P') obtained by multiplying ) by the rotation matrix R 3d Can determine ) (P' 3d= RP 3d ). The refrigerator (1000) is in a rotated three-dimensional position (P' 3d ) can be projected onto 2D screen coordinates (P'2d = KP' 3d The refrigerator (1000) can find the top left point and the bottom right point among the four points (P'2d) projected into 2D screen coordinates. In this case, the refrigerator (1000) can accurately find the refrigerator door area in the front image of the refrigerator door. The refrigerator (1000) can crop the refrigerator door area in the front image of the refrigerator door using the top left point and the bottom right point to generate a pure front image of the refrigerator door that contains almost only the refrigerator door area.

[0162] FIG. 10 is a drawing for illustrating a front view of a refrigerator door according to one embodiment of the present disclosure.

[0163] Referring to FIG. 10, the refrigerator (1000) can obtain a front view of the refrigerator door (1020) by rotating the refrigerator door image (1010) captured with the refrigerator door open in three dimensions and cropping only the refrigerator door area. The front view of the refrigerator door (1020) can have improved visibility compared to the refrigerator door image (1010). Therefore, when the refrigerator (1000) provides the user with the front view of the refrigerator door (1020) instead of the refrigerator door image (1010), the user can easily identify the food stored in the refrigerator door.

[0164] However, since the front view of the refrigerator door (1020) obtained using a vanishing point is a warped version of the refrigerator door image (1010) captured using a camera (1400) located at the top center of the refrigerator body, the proportions of the objects included in the front view of the refrigerator door (1020) differ from the proportions of the actual products included in the image (1030) taken from the actual front of the refrigerator door, so the front view of the refrigerator door (1020) may appear unnatural. In particular, the front view of the refrigerator door (1020) obtained using a vanishing point differs from the image (1030) taken from the actual front of the refrigerator door as it goes down. For example, an object stored in the door bin at the very bottom of the front view of the refrigerator door (1020) may appear unnaturally elongated. Therefore, it is necessary to naturally adjust the proportions of the objects included in the front view of the refrigerator door (1020). We will examine in detail the operation of the refrigerator (1000) adjusting the proportions of the objects included in the front view of the refrigerator door (1020) with reference to FIG. 11.

[0165] FIG. 11 is a drawing for explaining the operation of obtaining a scale adjustment ratio corresponding to each of the zones of a plurality of door bins according to one embodiment of the present disclosure.

[0166] Referring to 1101 in FIG. 11, the refrigerator (1000) can separate multiple door bin areas from the front image of the refrigerator door obtained using a vanishing point in order to adjust the front image of the refrigerator door to a ratio that makes it look as natural as possible.

[0167] According to one embodiment of the present disclosure, a refrigerator (1000) can detect at least one horizontal line in a front image of the refrigerator door. For example, the refrigerator (1000) can detect a horizontal line under each door bin. The refrigerator (1000) can separate a plurality of door bin zones based on at least one horizontal line. For example, if the refrigerator (1000) detects a first horizontal line under a first door bin and a second horizontal line under a second door bin in a front image of the refrigerator door, it can distinguish a first door bin zone (111), a second door bin zone (1112), and a third door bin zone (1113) based on the first horizontal line and the second horizontal line.

[0168] Referring to 1102 in FIG. 11, according to one embodiment of the present disclosure, a refrigerator (1000) can determine the scale adjustment ratio of each door bin area by using the position and direction of the camera (1400). 1102 in FIG. 11 shows a side view of the refrigerator door. The refrigerator (1000) can determine the ratio (B / A) between a second line (A) perpendicular to a first line passing through the center point of each door bin area from the camera (1400) and a third line (B) projected from the second line (A) with respect to the front as the vertical scale adjustment ratio of each of the multiple door bin areas. That is, if the angle (θ) between the second line (A) and the third line (B) is calculated, Cosθ can be the vertical scale adjustment ratio.

[0169] Since the camera (1400) is positioned at the top center of the refrigerator body (1000), the angle between the first line and the second line can increase as it goes toward the bottom of the refrigerator door. Therefore, the refrigerator (1000) can determine the vertical scale adjustment ratio of the bottom door bin area to be larger than the vertical scale adjustment ratio of the top door bin area. For example, the refrigerator (1000) can determine the scale adjustment ratio of the first door bin area (1111) as Cos16°=0.96, the scale adjustment ratio of the second door bin area (1112) as Cos43°=0.73, and the scale adjustment ratio of the third door bin area (1113) as Cos59°=0.52.

[0170] According to one embodiment of the present disclosure, the refrigerator (1000) can adjust the vertical length of each door bin area by applying a scale adjustment ratio to each door bin area. For example, if the vertical scale adjustment ratio of the first door bin area (1111) is 0.96 and the vertical length of the first door bin area (1111) is 340 pixels, the refrigerator (1000) can reduce the vertical length of the first door bin area (1111) to 326 pixels. If the vertical scale adjustment ratio of the second door bin area (1112) is 0.73 and the vertical length of the second door bin area (1112) is 560 pixels, the refrigerator (1000) can reduce the vertical length of the second door bin area (1112) to 409 pixels. If the vertical scale adjustment ratio of the third door bin area (1113) is 0.52 and the vertical length of the third door bin area (1113) is 850 pixels, the refrigerator (1000) can reduce the vertical length of the third door bin area (1113) to 442 pixels.

[0171] The refrigerator (1000) can provide the user with a front view of a natural shape with scaled-up door bin area. Refer to FIG. 12.

[0172] FIG. 12 is a drawing for illustrating a front view of a refrigerator door with the scale adjusted for each of the zones of a plurality of door bins according to one embodiment of the present disclosure.

[0173] The refrigerator (1000) can obtain a front view of the refrigerator door (1020) by three-dimensionally rotating a refrigerator door image (1010) taken with the refrigerator door open and cropping only the refrigerator door area. The refrigerator (1000) can determine the scale adjustment ratio of each door bin area and display a scaled front view of the refrigerator door (1201) of the first door bin area (1111), the second door bin area (1112), and the third door bin area (1113) based on the scale adjustment ratio of each door bin area. The front view of the refrigerator door (1020) can have improved visibility compared to the refrigerator door image (1010), and the scaled front view of the refrigerator door (1201) can naturally express the size of the object compared to the front view of the refrigerator door (1020) before scale adjustment.

[0174] However, the scaled-up front view of the refrigerator door (1201) differs from the actual front view (1030) in that there is no background area within each door bin area. Therefore, in order to implement it like the actual front view (1030), it is necessary to generate a background image within each door bin area. The operation of the refrigerator (1000) generating a background image within each door bin area will be examined in detail later with reference to FIG. 14.

[0175] FIG. 13 is a drawing for explaining the operation of a refrigerator (1000) according to one embodiment of the present disclosure acquiring a refrigerator door image at a different angle based on a front image of the refrigerator door.

[0176] According to one embodiment of the present disclosure, the refrigerator (1000) can generate an image of the refrigerator door at an angle other than the front. The refrigerator (1000) can generate an image of the refrigerator door at a different angle (1302) based on a front image (1301) of the refrigerator door. For example, when creating an image of the entire refrigerator, the refrigerator (1000) can convert the front image (1301) of the refrigerator door into an image of the refrigerator door at a specific angle (1302).

[0177] According to one embodiment of the present disclosure, a refrigerator (1000) can convert a front image of the refrigerator door into a refrigerator door image at a different angle using a rotation matrix. For example, the refrigerator (1000) can determine a rotation angle based on the x, y, and z axes, and calculate a 3D rotation matrix based on the rotation angle values ​​to rotate the front image of the refrigerator door.

[0178] According to one embodiment of the present disclosure, a refrigerator (1000) can convert a refrigerator door image at a different angle using homography. For example, the refrigerator (1000) can convert a refrigerator door image at a different angle by calculating homography based on the four vertices of a frontal image of the refrigerator door and the 2D position values ​​of the corresponding four vertices after conversion. Homography may be a transformation matrix that performs a perspective transformation from a point on one plane to a point on another plane.

[0179] FIG. 14 is a flowchart for explaining a method for displaying a front view of a refrigerator door including a background image corresponding to each of the zones of a plurality of door bins according to one embodiment of the present disclosure.

[0180] Referring to FIG. 14, a method for a refrigerator (1000) to display a front image of a refrigerator door including a background image corresponding to each of the zones of a plurality of door bins may include steps S1410 to S1430. In one embodiment of the present disclosure, steps S1410 to S1430 may be executed by at least one processor included in the refrigerator (1000). A method for a refrigerator (1000) to display a front image of a refrigerator door including a background image corresponding to each of the zones of a plurality of door bins is not limited to that illustrated in FIG. 14, and in one or more embodiments, additional steps not illustrated in FIG. 14 may be included, or some steps may be omitted.

[0181] In step S1410, a refrigerator (1000) according to one embodiment of the present disclosure can obtain mask area information for filling a background area corresponding to each of the areas of a plurality of door bins.

[0182] According to one embodiment of the present disclosure, a refrigerator (1000) can obtain mask area information for filling a background area corresponding to each of the sections of a plurality of door bins when each of the sections of a plurality of door bins is reduced in the vertical direction based on a scale adjustment ratio. At this time, the mask area information may include location information of the mask area corresponding to each door bin section and size information of the mask area corresponding to each door bin section.

[0183] Generally, since the objects stored in the door bin (e.g., beverages or sauces) are lower in height than the door bin, there may be an empty space (background area) in the upper area of ​​the door bin. Therefore, the refrigerator (1000) can determine the upper area of ​​each door bin zone as a mask area. The refrigerator (1000) can also determine the size of the mask area corresponding to each door bin zone based on the difference between the size of each door bin zone before reduction and the size of each door bin zone after reduction. Since the reduction ratio is greater for the lower door bin zones, the size of the mask area may be larger. The operation of the refrigerator (1000) determining the size of the mask area corresponding to each door bin zone will be examined in detail later with reference to FIG. 15.

[0184] In step S1420, a refrigerator (1000) according to one embodiment of the present disclosure can obtain a background image corresponding to each of the areas of the plurality of door bins by applying mask area information and an image of each of the areas of the plurality of door bins to a generative AI (Artificial Intelligence) model.

[0185] According to one embodiment of the present disclosure, a generative AI model may be an AI model capable of generating new content from a given input. For example, the generative AI model may be an AI model trained to generate a background image corresponding to each of the areas of a plurality of door beans.

[0186] According to one embodiment of the present disclosure, a refrigerator (1000) may input an image of an empty refrigerator door, mask area information of each door bin area (e.g., location information of the mask area, size information of the mask area), and an image of each door bin area included in a scaled front image of the refrigerator door (reference image). For example, the refrigerator (1000) may input an image of an empty refrigerator door, an image of a first door bin area, mask area information of the first door bin area, an image of a second door bin area, mask area information of the second door bin area, an image of a third door bin area, and mask area information of the third door bin area into the generative AI model. Additionally, the refrigerator (1000) may provide a prompt to the generative AI model to "naturally fill the upper mask area while maintaining the image of each door bin area (reference image)."

[0187] Based on the prompt, the generative AI model can generate a background image in the top mask area while preserving the video of each doorbin area. In other words, the generative AI model can generate and output a video of each doorbin area that includes a background image at the top.

[0188] According to one embodiment of the present disclosure, the refrigerator (1000) may additionally input information regarding the internal lighting position and door opening angle of the refrigerator (1000) into a generative AI model. Additionally, the refrigerator (1000) may provide a prompt to the generative AI model to generate a natural shadow in the upper mask area. In this case, the generative AI model may output a background image including a shadow for each door bin area.

[0189] In step S1430, a refrigerator (1000) according to one embodiment of the present disclosure may display a front image of a refrigerator door including a background image corresponding to each of the areas of a plurality of door bins.

[0190] According to one embodiment of the present disclosure, a refrigerator (1000) may display a front view of the refrigerator door containing a background image corresponding to each of the sections of a plurality of door bins on a user terminal (e.g., a smartphone) connected via a display of the refrigerator (1000) or a server (not shown). By displaying a front view of the refrigerator door containing a background image corresponding to each of the sections of the plurality of door bins, the refrigerator (1000) may provide a natural refrigerator door image to the user. By checking the front view of the refrigerator door displayed on the display of the refrigerator (1000) or the user terminal, the user can accurately identify the food stored in the refrigerator door without opening the refrigerator door.

[0191] FIG. 15 is a drawing for explaining the operation of a refrigerator (1000) according to one embodiment of the present disclosure determining the size of a mask area.

[0192] According to one embodiment of the present disclosure, the refrigerator (1000) can determine the size of a mask area corresponding to each door bin area based on the difference between the size of each door bin area before reduction and the size of each door bin area after reduction.

[0193] For example, if the total vertical length of the first door bean area (1511) is 340 pixels and the vertical scaling ratio of the first door bean area (1511) is 0.96, the vertical length of the reduced first door bean area (1510) may be 326 pixels. At this time, the refrigerator (1000) may determine the size of the first mask area (1501) by the difference between the total vertical length of the first door bean area (1511) before reduction and the vertical length of the reduced first door bean area (1510). For example, the refrigerator (1000) may determine 14 pixels (=340*0.04) as the size of the first mask area (1501).

[0194] If the total vertical length of the second door bin area (1512) is 560 pixels and the vertical scale adjustment ratio of the second door bin area (1512) is 0.73, the vertical length of the reduced second door bin area (1520) may be 409 pixels. At this time, the refrigerator (1000) may determine the size of the second mask area (1502) by the difference between the total vertical length of the second door bin area (1512) before reduction and the vertical length of the reduced second door bin area (1520). For example, the refrigerator (1000) may determine 151 pixels (= 560 * 0.27) as the size of the second mask area (1502).

[0195] If the total vertical length of the third door bin area (1513) is 850 pixels and the vertical scale adjustment ratio of the third door bin area (1513) is 0.52, the vertical length of the reduced third door bin area (1530) may be 442 pixels. At this time, the refrigerator (1000) may determine the size of the third mask area (1503) by the difference between the total vertical length of the third door bin area (1513) before reduction and the vertical length of the reduced third door bin area (1530). For example, the refrigerator (1000) may determine 408 pixels (= 850 * 0.48) as the size of the third mask area (1503).

[0196] FIG. 16 is a diagram illustrating the operation of applying mask area information and images of each of the areas of a plurality of doorbins to a generative AI model according to one embodiment of the present disclosure.

[0197] Referring to FIG. 16, the refrigerator (1000) can input an image (1611) of an empty refrigerator door, an image of a reduced first door bin area (1510), information on a first mask area (1501) above the reduced first door bin area (1510) (e.g., location and size of the first mask area (1501)), an image of a reduced second door bin area (1520), information on a second mask area (1502) above the reduced second door bin area (1520) (e.g., location and size of the second mask area (1502)), an image of a reduced third door bin area (1530), and information on a third mask area (1503) above the reduced third door bin area (1530) (e.g., location and size of the third mask area (1503)) into a generative AI model (1612). Additionally, the refrigerator (1000) can provide a prompt to the generative AI model (1612) to "naturally fill the upper mask area while maintaining the image (reference image) of each door bin area."

[0198] A generative AI model (1612) can divide the image (1611) of an empty refrigerator door into a reduced first door bin area (1510), a first mask area (1501), a reduced second door bin area (1520), a second mask area (1502), a reduced third door bin area (1530), and a third mask area (1503) based on the image (1611) of the empty refrigerator door. The generative AI model (1612) can fill the reduced first door bin area (1510) with the image of the reduced first door bin area (1510) and generate a natural first background image (1601) that matches the image of the reduced first door bin area (1510) in the first mask area (1501). The generative AI model (1612) can fill the reduced second doorbin area (1520) with an image of the reduced second doorbin area (1520) and generate a natural second background image (1602) that matches the image of the reduced second doorbin area (1520) in the second mask area (1502). The generative AI model (1612) can fill the reduced third doorbin area (1530) with an image of the reduced third doorbin area (1530) and generate a natural third background image (1603) that matches the image of the reduced third doorbin area (1530) in the third mask area (1503). For example, if the image of the third doorbin area (1530) contains a beverage can with a missing top portion, the third background image (1603) may generate and include a missing top portion.

[0199] The generative AI model (1612) can output a front image of the refrigerator door including an image of the reduced first door bin area (1510) and a first background image (1601), an image of the reduced second door bin area (1520) and a second background image (1602), and an image of the reduced third door bin area (1530) and a third background image (1603).

[0200] Meanwhile, according to one embodiment of the present disclosure, the generative AI model (1612) may first output a first doorbin image combined with an image of a reduced first doorbin area (1510) and a first background image (1601), then output a second doorbin image combined with an image of a reduced second doorbin area (1520) and a second background image (1602), and finally output a third doorbin image combined with an image of a reduced third doorbin area (1530) and a third background image (1603). In this case, the refrigerator (1000) may generate or correct a front image of the refrigerator door using the first doorbin image, the second doorbin image, and the third doorbin image output from the generative AI model (1612).

[0201] FIG. 17 is a drawing for illustrating a front view of a refrigerator door including a background image corresponding to each of the zones of a plurality of door bins according to one embodiment of the present disclosure.

[0202] The refrigerator (1000) can obtain a front view of the refrigerator door (1710) by using a camera (1400) at the top center of the main body to capture a video of the refrigerator door while the refrigerator door is open, rotating it in three dimensions, and then cropping only the refrigerator door area. The refrigerator (1000) can determine the scale adjustment ratio of each door bin area and generate a scaled front view of the refrigerator door (1720) of the first door bin area (1711), the second door bin area (1712), and the third door bin area (1713) based on the scale adjustment ratio of each door bin area. The refrigerator (1000) can apply the front view of the refrigerator door (1720) to a generative AI model to display a front view of the refrigerator door (1730) that includes a background image of each door bin area.

[0203] The front view (1730) of the refrigerator door containing the background image of each door bin area may be similar in proportion and shape to an actual image of the refrigerator door taken from the front, so it can provide the user with a natural front view (1730) of the refrigerator door. The user can check the food stored in the refrigerator door using the front view (1730) of the refrigerator door without opening the refrigerator door.

[0204] FIG. 18 is a drawing for explaining the operation of generating a background image including a shadow in a mask area according to one embodiment of the present disclosure.

[0205] According to one embodiment of the present disclosure, a refrigerator (1000) may input to a generative AI model (1800) an image of an empty refrigerator door, mask area information of each door bin area (e.g., location information of the mask area, size information of the mask area), an image of each door bin area included in a scaled front image of the refrigerator door (reference image), the internal lighting position of the refrigerator (1000), and door opening angle information. The door opening angle information may be obtained in the manner described in FIG. 3 or FIG. 4. The refrigerator (1000) may provide a prompt to the generative AI model (1800) to "generate a background image that includes natural shadows in the mask area while naturally filling the top mask area while maintaining the image of each door bin area (reference image), taking into account the refrigerator internal lighting position and the door opening angle." The generative AI model (1800) may be an AI model trained to generate a background image that includes shadows.

[0206] For example, the generative AI model (1800) may receive input regarding the image (1811) of the first doorbin area, the image (1812) of the second doorbin area, and the second mask area (1810) at the top of the second doorbin area (e.g., location and size), internal lighting location information, door opening angle information, and a prompt (e.g., creating a background image that naturally fills the top mask area while maintaining the image (reference image) of each doorbin area, while including natural shadows in the mask area by considering the internal lighting location of the refrigerator (1000) and the door opening angle). In this case, the generative AI model (1800) may create a background image (1820) including shadows that can be generated by objects stored in the first doorbin area by considering the internal lighting location of the refrigerator (1000), the door opening angle, and the objects included in the image (1811) of the first doorbin area. And the generative AI model (1800) can fill the second mask area (1810) with a background image (1820) that includes a shadow. In this case, the refrigerator (1000) can provide the user with a front view of the refrigerator door in a more natural shape.

[0207] FIG. 19 is a block diagram for explaining the function of a refrigerator (1000) according to one embodiment of the present disclosure.

[0208] FIG. 20 is a detailed block diagram for explaining the function of a refrigerator (1000) according to one embodiment of the present disclosure.

[0209] As illustrated in FIG. 19, a refrigerator (1000) according to one embodiment of the present disclosure may include a camera (1400) and a control unit (1200) (i.e., a processor (1210) and a memory (1220)). However, not all components illustrated in FIG. 19 are essential components. The refrigerator (1000) may be implemented with more components than those illustrated in FIG. 19, or with fewer components. For example, as illustrated in FIG. 20, a refrigerator (1000) according to one embodiment of the present disclosure may include a cooling unit (1100), a control unit (1200), a display unit (1300), a camera (1400), a sensor unit (1500), a voice unit (1600), and a communication interface (1700).

[0210] The above components will be examined in turn below.

[0211] The cooling unit (1100) may include a compressor (1110), a condenser (1120), an expander (1130), and an evaporator (1140).

[0212] The compressor (1110) compresses the refrigerant to a high temperature and high pressure state. The compressor (1110) can compress the gaseous refrigerant to a high temperature and high pressure state by receiving electrical energy from an external source and utilizing rotational force such as an electric motor. The compressor (1110) is connected to the condenser (1120) and can move the compressed refrigerant to the condenser (1120). The compressor (1110) compresses the refrigerant and pushes it into the condenser (1120), thereby operating a refrigeration cycle of compression, condensation, expansion, and evaporation. Therefore, when the compressor (1110) is operated, the cold air generated in the evaporator (1140) is supplied to the storage room.

[0213] The condenser (1120) condenses the high-temperature, high-pressure refrigerant compressed from the compressor (1110). The condenser (1120) dissipates the heat generated while condensing the refrigerant. The refrigerant condensed while passing through the condenser (1120) is moved to the expansion valve (1130). The refrigerant condensed in the condenser (1802) becomes a low-temperature, low-pressure liquid state while passing through the expansion valve (1130). The liquid refrigerant passes through the expansion valve (1130) and is moved to the evaporator (1140).

[0214] The evaporator (1140) evaporates the low-temperature, low-pressure liquid refrigerant that has passed through the expansion valve. As the liquid refrigerant evaporates, heat exchange with the surrounding gas takes place in the evaporator (1140). As the liquid refrigerant evaporates, it absorbs latent heat from the surroundings, and as a result, the gas surrounding the evaporator (1140) is cooled, generating cold air. The completely evaporated refrigerant is supplied back to the compressor (1110), and the cooling cycle circulates. A heater may be provided around the evaporator (1140) to remove frost formed on the evaporator (1140).

[0215] The control unit (1200) may include a memory (1220) for storing or remembering a program and / or data for controlling a refrigerator (1000), and a processor (1210) for outputting a control signal for controlling a cooling unit (1100), etc., according to the program and / or data stored in the memory (1220).

[0216] The processor (1210) included in the refrigerator (1000) may be one or multiple. The processor (1210) may include at least one of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), and NPU (Neural Processing Unit). At least one processor (1210) may be implemented in the form of an integrated system-on-chip (SoC) including one or more electronic components. Each of the at least one processor (1210) may be implemented as separate hardware (H / W). At least one processor (1210) may be represented as a MICOM (Micro-Computer, Microprocessor Computer, Microprocessor controller), MPU (Micro Processor unit), or MCU (Micro Controller Unit).

[0217] At least one processor (1210) according to the present disclosure may be implemented as a single core processor or as a multicore processor.

[0218] The memory (1220) stores or records various information, data, commands, programs, etc., necessary for the operation of the refrigerator (1000). The memory (1220) can store temporary data that occurs while generating control signals to control components included in the refrigerator (1000). The memory (1220) may include at least one of volatile memory or non-volatile memory, or a combination thereof.

[0219] The memory (1220) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Programs stored in the memory (1220) may be classified into multiple modules according to their functions.

[0220] The display unit (1300) is intended to output a video signal. The display unit (1300) may include a display panel (1310) and a touch panel (1320). When the display panel (1310) and the touch panel (1320) form a layered structure and are configured as a touchscreen, the display unit (1300) may be used as an input device in addition to an output device.

[0221] The display unit (1300) may include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display. Additionally, depending on the implementation form of the refrigerator (1000), the refrigerator (1000) may include two or more display units (1300).

[0222] According to one embodiment of the present disclosure, the display unit (1300) may include an LED display (1330). The LED display (1330) may flash in a predetermined color (e.g., red, blue, etc.) when a notification message is output as voice through a speaker (1620).

[0223] The camera (1400) may include an external camera that captures the external environment and / or an internal camera that captures the internal environment. There may be multiple internal cameras that capture the internal environment. For example, the internal cameras may include a main camera located at the top center of the refrigerator (1000) and sub-cameras located in each storage compartment. The camera (1400) may include at least one of an RGB camera, a depth camera, or a thermal imaging camera.

[0224] The sensor unit (1500) may include at least one of an environment sensor, a proximity sensor, a door opening / closing detection sensor, and a spectroscopic sensor, but is not limited thereto. The environment sensor is a sensor for acquiring environmental information inside the refrigerator (1000) and may include at least one of an odor sensor, a temperature sensor, and a humidity sensor. Since the function of each sensor can be intuitively inferred by a person skilled in the art from its name, a detailed description will be omitted.

[0225] The voice unit (1600) may include a microphone (1610) and a speaker (1620).

[0226] The microphone (1610) receives an external acoustic signal and processes it into electrical voice data. For example, the microphone (1610) can receive an acoustic signal (e.g., a voice command) from an external device or a speaker. The microphone (1610) can use various noise removal algorithms to remove noise generated during the process of receiving the external acoustic signal.

[0227] The speaker (1620) outputs audio data received from the communication interface (1700) or stored in the memory (1220). Additionally, the speaker (1620) outputs an acoustic signal related to a function performed in the refrigerator (1000) (e.g., message reception sound, notification sound).

[0228] The communication interface (1700) may include a short-range communication unit, a long-range communication unit, etc. The short-range wireless communication unit may include a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a Near Field Communication (NFC) unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an Infrared Data Association (IrDA) communication unit, a WFD (Wi-Fi Direct) communication unit, an UWB (ultra wideband) communication unit, an Ant+ communication unit, etc., but is not limited thereto. The long-range communication unit may be used for the refrigerator (1000) to communicate remotely with an external server. The long-range communication unit may include the Internet, a computer network (e.g., LAN or WAN), and a mobile communication unit. The mobile communication unit may include a 3G module, a 4G module, a 5G module, an LTE module, an NB-IoT module, an LTE-M module, etc., but is not limited thereto.

[0229] FIG. 21 is a drawing for explaining a communication system of a refrigerator (1000) according to one embodiment of the present disclosure.

[0230] A refrigerator (1000) according to one embodiment of the present disclosure can communicate with an external server (2000), a user terminal (3000), and an external device (4000) through a network.

[0231] The external server (2000) may include a communication module capable of communicating with another server, a refrigerator (1000), an external device (4000), or a user terminal (3000), at least one processor capable of processing data received from another server, a refrigerator (1000), an external device (4000), or a user terminal (3000), and at least one memory capable of storing a program for processing data or processed data. This external server (2000) may be implemented as various computing devices such as a workstation, a cloud, a data drive, or a data station. The external server (2000) may be implemented as one or more servers physically or logically separated based on functions, detailed configurations of functions, or data, and may transmit and receive data and process the transmitted and received data through communication between each server.

[0232] The external server (2000) can perform functions such as managing user accounts, registering refrigerators (1000) and external devices (4000) associated with user accounts, and managing or controlling the registered refrigerators (1000) and external devices (4000). For example, a user can create a user account by accessing the external server (2000) through a user terminal (3000). The user account can be identified by an ID and password set by the user. The external server (2000) can register refrigerators (1000) and external devices (4000) to the user account according to a set procedure. For example, the external server (2000) can register, manage, and control refrigerators (1000) and external devices (4000) by linking identification information (e.g., serial number or MAC address, etc.) of refrigerators (1000) and external devices (4000) to the user account.

[0233] A user terminal (3000) may include a communication module capable of communicating with a refrigerator (1000), an external device (4000), or an external server (2000), a user interface that receives user input or outputs information to the user, at least one processor that controls the operation of the user terminal (3000), and at least one memory in which a program for controlling the operation of the user terminal (3000) is stored.

[0234] The user terminal (3000) may be carried by the user or placed in the user's home or office, etc. The user terminal (3000) may include, but is not limited to, a personal computer, terminal, portable telephone, smartphone, handheld device, wearable device, etc.

[0235] A program, i.e., an application, for controlling the refrigerator (1000) and external device (4000) can be stored in the memory of the user terminal (3000). The application may be sold with the user terminal (3000) already installed, or it may be downloaded and installed from an external server.

[0236] By running an application installed on a user terminal (3000), the user can connect to an external server (2000) to create a user account, and communicate with the external server (2000) based on the logged-in user account to register a refrigerator (1000) and an external device (4000).

[0237] For example, if the refrigerator (1000) and external device (4000) are operated in accordance with the procedure guided by the application installed on the user terminal (3000) so that the refrigerator (1000) and external device (4000) can be connected to the external server (2000), the refrigerator (1000) and external device (4000) can be registered to the user account by registering the identification information (e.g., serial number or MAC address, etc.) of the refrigerator (1000) and external device (4000) to the corresponding user account on the external server (2000).

[0238] The user can control the refrigerator (1000) and external device (4000) using an application installed on the user terminal (3000). For example, when the user logs into a user account using an application installed on the user terminal (3000), the refrigerator (1000) and external device (4000) registered to the user account appear, and when a control command for the refrigerator (1000) or external device (4000) is entered, the control command can be transmitted to the refrigerator (1000) or external device (4000) through an external server (2000).

[0239] The external device (4000) may include a communication module capable of communicating with a refrigerator (1000), a user terminal (3000), or an external server (2000), a user interface that receives user input or outputs information to the user, at least one processor that controls the operation of the external device (4000), and at least one memory that stores a program for controlling the operation of the external device (4000).

[0240] The external device (4000) may be at least one of various types of home appliances. For example, the external device (4000) may include at least one of a dishwasher, an electric range, an electric oven, an air conditioner, a garment care system, a washing machine, a dryer, a microwave oven, an air purifier, a robot vacuum cleaner, a vacuum cleaner, and a television, but is not limited thereto.

[0241] A network may include both wired and wireless networks. Wired networks include cable networks or telephone networks, etc., and wireless networks may include all networks that transmit and receive signals via radio waves. Wired and wireless networks may be connected to each other.

[0242] Networks may include wide area networks (WANs) such as the Internet, local area networks (LANs) formed around access points (APs), and short-range wireless networks that do not pass through access points (APs). Short-range wireless networks may include Bluetooth (IEEE 802.15.1), Zigbee (IEEE 802.15.4), Wi-Fi Direct, Near Field Communication (NFC), Z-Wave, etc., but are not limited thereto.

[0243] An access point (AP) can connect a refrigerator (1000), an external device (4000), or a user terminal (3000) to a wide area network (WAN) to which an external server (2000) is connected. The refrigerator (1000), the external device (4000), or the user terminal (3000) can be connected to the external server (2000) through the wide area network (WAN).

[0244] The access point (AP) can communicate with a refrigerator (1000), an external device (4000), or a user terminal (3000) using wireless communication such as Wi-Fi (IEEE 802.11), Bluetooth (IEEE 802.15.1), or Zigbee (IEEE 802.15.4), and can connect to a wide area network (WAN) using wired communication, but is not limited thereto.

[0245] According to one embodiment of the present disclosure, the refrigerator (1000) may be directly connected to a user terminal (3000), an external device (4000), or an external server (2000) without going through an access relay (AP).

[0246] The refrigerator (1000) can be connected to an external device (4000), a user terminal (3000), or an external server (2000) via a long-range wireless network or a short-range wireless network. For example, the refrigerator (1000) can be connected to a user terminal (3000) via a short-range wireless network (e.g., Wi-Fi Direct).

[0247] The refrigerator (1000) may be connected to a user terminal (3000), an external server (2000), or an external device (4000) via a wide area network (WAN) using a long-distance wireless network (e.g., a cellular communication module). Additionally, the refrigerator (1000) may be connected to a wide area network (WAN) using wired communication and connected to a user terminal (3000), an external server (2000), or an external device (4000) via the wide area network (WAN).

[0248] If the refrigerator (1000) can connect to a wide area network (WAN) using wired communication, it may operate as a connection relay. Accordingly, the refrigerator (1000) can connect an external device (4000) to a wide area network (WAN) to which an external server (2000) is connected. Additionally, the external device (4000) can connect the refrigerator (1000) to a wide area network (WAN) to which an external server (2000) is connected.

[0249] The refrigerator (1000) can transmit information regarding operation or status to an external device (4000), a user terminal (3000), or an external server (2000) via a network. For example, the refrigerator (1000) can transmit information regarding operation or status to the external device (4000), a user terminal (3000), or an external server (2000) when a request is received from the external server (2000), when a specific event occurs in the refrigerator (1000), or periodically or in real time. When the external server (2000) receives information regarding operation or status from the refrigerator (1000), it can update the stored information regarding operation or status of the refrigerator (1000) and transmit the updated information regarding operation and status of the refrigerator (1000) to the user terminal (3000) via a network. Here, updating information may include various operations that change existing information, such as adding new information to existing information or replacing existing information with new information.

[0250] The refrigerator (1000) can obtain various information from an external device (4000), a user terminal (3000), or an external server (2000), and provide the obtained information to the user. For example, the refrigerator (1000) can obtain information related to the functions of the refrigerator (1000) (e.g., recipes, etc.) and various environmental information (e.g., weather, temperature, humidity, etc.) from the external server (2000), and can output the obtained information through a user interface.

[0251] The refrigerator (1000) can operate according to control commands received from an external device (4000), a user terminal (3000), or an external server (2000). For example, if the refrigerator (1000) has obtained prior approval from a user to operate according to control commands from an external server (2000) even without user input, the refrigerator (1000) can operate according to control commands received from the external server (2000). Here, the control commands received from the external server (2000) may include, but are not limited to, control commands entered by the user through the user terminal (3000) or control commands based on pre-set conditions.

[0252] The user terminal (3000) can transmit information about the user to the refrigerator (1000), external device (4000), or external server (2000) through a communication module. For example, the user terminal (3000) can transmit information about the user's location, health status, preferences, schedule, etc., to the external server (2000). The user terminal (3000) can transmit information about the user to the refrigerator (1000) or external server (2000) based on the user's prior approval.

[0253] A refrigerator (1000), an external device (4000), a user terminal (3000), or an external server (2000) may determine control commands using technology such as artificial intelligence. For example, the external server (2000) may receive information regarding the operation or status of the refrigerator (1000) and the external device (4000), or receive information regarding the user of the user terminal (3000), process it using technology such as artificial intelligence, and transmit the processing result or control command to the refrigerator (1000), the external device (4000), or the user terminal (3000) based on the processing result.

[0254] A method for a refrigerator (1000) to provide a refrigerator door image according to one embodiment of the present disclosure may include: a step of detecting a plurality of lines in a refrigerator door image including a plurality of door bins obtained using a camera (1400) located inside the main body of the refrigerator (1000); a step of obtaining a pose of the camera (1400) using the detected plurality of lines; a step of obtaining a front image of the refrigerator door by converting a viewpoint corresponding to the pose of the camera (1400) to a viewpoint in which the camera (1400) looks directly at the refrigerator door; a step of separating the areas of the plurality of door bins included in the front image of the refrigerator door; a step of obtaining a scale adjustment ratio corresponding to each of the areas of the plurality of door bins based on the pose of the camera (1400); and a step of displaying a front image of the refrigerator door in which the scale of each of the areas of the plurality of door bins is adjusted based on the scale adjustment ratio.

[0255] A step of detecting a plurality of lines according to one embodiment of the present disclosure may include: a step of performing distortion correction on a refrigerator door image; and a step of detecting a plurality of lines in a refrigerator door image in which distortion correction has been performed.

[0256] A step of performing distortion correction for a refrigerator door image according to one embodiment of the present disclosure may include: a step of detecting straight line components in a refrigerator door image; a step of estimating the open angle of the refrigerator door based on the straight line components; and a step of performing distortion correction for the refrigerator door image when the open angle of the refrigerator door is within a predetermined angle range.

[0257] A step of detecting a plurality of lines according to one embodiment of the present disclosure may include a step of detecting a plurality of lines within defined regions of interest.

[0258] The step of determining the pose of a camera (1400) according to one embodiment of the present disclosure may include the step of determining a horizontal vanishing point using a plurality of lines.

[0259] A step of acquiring a front image of a refrigerator door according to one embodiment of the present disclosure may include: a step of determining a vertical vanishing point and a depth vanishing point using a horizontal vanishing point; and a step of determining a rotation matrix using the horizontal vanishing point, the vertical vanishing point, and the depth vanishing point.

[0260] A step of acquiring a front image of a refrigerator door according to one embodiment of the present disclosure may include: a step of rotating a refrigerator door image in three dimensions using a rotation matrix; and a step of acquiring a front image of a refrigerator door by cropping a refrigerator door area from a three-dimensionally rotated refrigerator door image.

[0261] A step of separating a plurality of door bin areas included in a front image of a refrigerator door according to one embodiment of the present disclosure may include: a step of detecting at least one horizontal line in a front image of a refrigerator door; and a step of separating a plurality of door bin areas based on at least one horizontal line.

[0262] The step of obtaining a scale adjustment ratio according to one embodiment of the present disclosure may include determining the ratio between a second line perpendicular to a first line passing through the center point of each of the areas of a plurality of doorbins from a camera (1400) and a third line projected with respect to the front as the vertical scale adjustment ratio of each of the areas of a plurality of doorbins.

[0263] A step of displaying a front image of a refrigerator door according to one embodiment of the present disclosure may include: a step of obtaining mask area information for filling a background area corresponding to each of the sections of a plurality of door bins when each of the sections of a plurality of door bins is reduced in the vertical direction based on a scale adjustment ratio; a step of applying the mask area information and the image of each of the sections of a plurality of door bins to a generative AI model to obtain a background image corresponding to each of the sections of a plurality of door bins; and a step of displaying a front image of a refrigerator door including a background image corresponding to each of the sections of a plurality of door bins.

[0264] The step of obtaining mask area information according to one embodiment of the present disclosure may include the step of determining the size of the mask area based on a scale adjustment ratio.

[0265] A step of acquiring a background image according to one embodiment of the present disclosure may include applying lighting position information inside the refrigerator and open angle information of the refrigerator door to a generative AI model to generate a background image including a shadow in a mask area.

[0266] A step of displaying a front image of a refrigerator door according to one embodiment of the present disclosure may include displaying a front image of the refrigerator door on a display (1300) of a refrigerator (1000) or on a user terminal (3000) connected via a server.

[0267] A refrigerator (1000) according to one embodiment of the present disclosure may include: a camera (1400) located inside a main body; a memory (1220) storing one or more instructions; and at least one processor (1210) operably coupled to the memory (1220) and including a processing circuit. By executing one or more instructions, either alone or in cooperation with at least one processor (2200), the refrigerator (1000) may detect a plurality of lines in a refrigerator door image including a plurality of door bins obtained using the camera (1400). By executing one or more instructions, either alone or in cooperation with at least one processor (2200), the refrigerator (1000) may obtain a pose of the camera (1400) using the detected plurality of lines. By having at least one processor (2200) execute one or more instructions, either alone or in cooperation, the refrigerator (1000) can acquire a front view of the refrigerator door by converting the viewpoint corresponding to the pose of the camera (1400) to a viewpoint where the camera (1400) faces the refrigerator door directly. By having at least one processor (2200) execute one or more instructions, either alone or in cooperation, the refrigerator (1000) can separate the areas of multiple door bins included in the front view of the refrigerator door. By having at least one processor (2200) execute one or more instructions, either alone or in cooperation, the refrigerator (1000) can acquire a scale adjustment ratio corresponding to each of the areas of multiple door bins based on the pose of the camera (1400). By having at least one processor (2200) execute one or more instructions, either alone or in cooperation, the refrigerator (1000) can display a front view of the refrigerator door in which the scale of each of the areas of multiple door bins is adjusted based on the scale adjustment ratio.

[0268] According to one embodiment of the present disclosure, by having at least one processor (2200) execute one or more instructions either alone or in cooperation, the refrigerator (1000) can perform distortion correction on the refrigerator door image. By having at least one processor (2200) execute one or more instructions either alone or in cooperation, the refrigerator (1000) can detect a plurality of lines in the refrigerator door image where distortion correction has been performed.

[0269] According to one embodiment of the present disclosure, by having at least one processor execute one or more instructions alone or in cooperation, the refrigerator (1000) can determine a horizontal vanishing point using a plurality of lines. By having at least one processor (2200) execute one or more instructions alone or in cooperation, the refrigerator (1000) can determine a vertical vanishing point and a depth vanishing point using the horizontal vanishing point. By having at least one processor (2200) execute one or more instructions alone or in cooperation, the refrigerator (1000) can determine a rotation matrix using the horizontal vanishing point, the vertical vanishing point, and the depth vanishing point.

[0270] According to one embodiment of the present disclosure, by having at least one processor (2200) execute one or more instructions either alone or in cooperation, the refrigerator (1000) can rotate the refrigerator door image in three dimensions using a rotation matrix. By having at least one processor (2200) execute one or more instructions either alone or in cooperation, the refrigerator (1000) can obtain a front view of the refrigerator door by cropping an area inside the refrigerator door from the three-dimensionally rotated refrigerator door image.

[0271] According to one embodiment of the present disclosure, by having at least one processor (2200) execute one or more instructions alone or in cooperation, the refrigerator (1000) can determine the ratio between a second line perpendicular to a first line passing through the center point of each of the areas of the plurality of door bins from a camera (1400) and a third line projected with respect to the front of the second line as the vertical scale adjustment ratio of each of the areas of the plurality of door bins.

[0272] According to one embodiment of the present disclosure, by executing one or more instructions by at least one processor (2200) alone or in cooperation, the refrigerator (1000) can obtain mask area information for filling a background area corresponding to each of the sections of the plurality of door bins when each of the sections of the plurality of door bins is reduced in the vertical direction based on a scaling ratio. By executing one or more instructions by at least one processor (2200) alone or in cooperation, the refrigerator (1000) can apply the mask area information and the image of each of the sections of the plurality of door bins to a generative AI model to generate a background image corresponding to each of the sections of the plurality of door bins. By executing one or more instructions by at least one processor (2200) alone or in cooperation, the refrigerator (1000) can display a front image of the refrigerator door containing a background image corresponding to each of the sections of the plurality of door bins.

[0273] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0274] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Claims

1. In a method in which a refrigerator (1000) provides a refrigerator door image, A step (S210) of detecting a plurality of lines in a refrigerator door image including a plurality of door bins obtained using a camera (1400) located inside the main body of the refrigerator; A step (S220) of obtaining the pose of the camera using the detected plurality of lines; A step (S230) of acquiring a frontal image of the refrigerator door by converting the viewpoint corresponding to the pose of the camera to a viewpoint in which the camera looks directly at the refrigerator door; Step (S240) of separating the areas of the plurality of door bins included in the front image of the refrigerator door; A step (S250) of obtaining a scale adjustment ratio corresponding to each of the zones of the plurality of doorbins based on the pose of the camera above; and A method comprising the step (S260) of displaying a front image of the refrigerator door in which the scale of each of the zones of the plurality of door bins is adjusted based on the scale adjustment ratio.

2. In claim 1, the step of detecting the plurality of lines is, A step of performing distortion correction on the above refrigerator door image; and A method comprising the step of detecting a plurality of lines in a refrigerator door image in which the above distortion correction has been performed.

3. In claim 1, the step of performing distortion correction on the refrigerator door image is: A step of detecting straight line components in the above refrigerator door image; A step of estimating the open angle of the refrigerator door based on the above linear components; and A method comprising the step of performing distortion correction on the refrigerator door image when the open angle of the refrigerator door is within a predetermined angle range.

4. In paragraph 2, the step of detecting the plurality of lines is, A method comprising the step of detecting multiple lines within a defined region of interest.

5. In paragraph 2, the step of determining the pose of the camera is, A method comprising the step of determining a horizontal vanishing point using the above plurality of lines.

6. In paragraph 5, the step of acquiring a front image of the refrigerator door is, A step of determining a vertical vanishing point and a depth vanishing point using the above horizontal vanishing point; and A method comprising the step of determining a rotation matrix using the above horizontal vanishing point, the above vertical vanishing point, and the above depth vanishing point.

7. In claim 6, the step of acquiring a front image of the refrigerator door is, A step of rotating the refrigerator door image in three dimensions using the above rotation matrix; and A method comprising the step of obtaining a front image of the refrigerator door by cropping the refrigerator door region from the above three-dimensional rotated refrigerator door image.

8. In any one of claims 1 to 7, the step of separating the areas of the plurality of door bins included in the front image of the refrigerator door is, A step of detecting at least one horizontal line in the front image of the refrigerator door; and A method comprising the step of separating the zones of the plurality of door bins based on at least one horizontal line.

9. In any one of claims 1 to 8, the step of obtaining the scale adjustment ratio is A method comprising the step of determining the ratio between a second line perpendicular to a first line passing through the center point of each of the areas of the plurality of doorbins from the camera and a third line projected with respect to the front as the vertical scale adjustment ratio of each of the areas of the plurality of doorbins.

10. In any one of claims 1 to 9, the step of displaying the front image of the refrigerator door is, When each of the areas of the plurality of doorbins is reduced in the vertical direction based on the above scale adjustment ratio, a step of obtaining mask area information for filling a background area corresponding to each of the areas of the plurality of doorbins; A step of applying the mask area information and the image of each of the regions of the plurality of doorbins to a generative AI model to obtain a background image corresponding to each of the regions of the plurality of doorbins; and A method comprising the step of displaying a front image of the refrigerator door including a background image corresponding to each of the zones of the plurality of door bins.

11. In paragraph 10, the step of acquiring the mask region information is, A method comprising the step of determining the size of the mask area based on the above-mentioned scale adjustment ratio.

12. In paragraph 10, the step of acquiring the background image is, A method comprising the step of applying lighting position information inside the refrigerator and open angle information of the refrigerator door to the generative AI model to generate the background image including a shadow in the mask area.

13. In any one of claims 1 to 12, the step of displaying the front image of the refrigerator door is: A method comprising the step of displaying a front image of the refrigerator door on a user terminal (3000) connected via a display (1300) of the refrigerator (1000) or a server (2000).

14. A camera (1400) located inside the main body; Memory (1220) for storing one or more instructions; and It includes at least one processor (1210) operably coupled to the memory and including a processing circuit, and By having at least one processor (1210) execute one or more instructions either alone or in cooperation, the refrigerator (1000) Detecting multiple lines in a refrigerator door image including multiple door bins obtained using the above camera (1400), and Using the above-detected plurality of lines, the pose of the camera (1400) is obtained, and By converting the viewpoint corresponding to the pose of the camera (1400) to a viewpoint where the camera (1400) looks directly at the refrigerator door, a frontal image of the refrigerator door is obtained. Separating the areas of the plurality of door bins included in the front image of the refrigerator door, and Based on the pose of the camera (1400), a scale adjustment ratio corresponding to each of the areas of the plurality of doorbins is obtained, and A refrigerator that displays a front image of the refrigerator door, with the scale of each of the zones of the plurality of door bins adjusted based on the above-mentioned scale adjustment ratio.

15. A computer-readable recording medium having a program recorded thereon for performing the method of any one of paragraphs 1 through 13 on a computer.

Images