Refrigerator and control method thereof

Abstract

According to an embodiment, a refrigerator comprises at least one storage compartment, at least one door configured to open and close the at least one storage compartment, at least one sensor, and at least one processor, wherein the at least one processor may be configured to acquire point cloud data by using the at least one sensor, to identify the gesture of a user, positioned within a first distance from the refrigerator, on the basis of the point cloud data, and to control a door of the at least one door to open on the basis of the identified gesture.

Description

Refrigerator and control method thereof

[0001] Various embodiments of the present disclosure relate to a refrigerator, and more specifically, to a refrigerator that automatically opens at least one door and a method for controlling said refrigerator.

[0002] A refrigerator is a device that uses a freezing cycle to store objects, such as food, in a fresh state, and may include a freezer compartment for storing objects at a sub-zero temperature and a refrigerator compartment for storing objects at a temperature above freezing.

[0003] Recently, with the proliferation of smart homes and the advancement of IoT technology, refrigerators can perform various innovative functions, such as voice control, remote operation via smartphone apps, and food management using internal cameras.

[0004] The refrigerator may include at least one door depending on the number or structure of the storage compartments. The refrigerator door can be used to load or unload objects from the storage compartments. When the refrigerator door is closed, it prevents cold air from the storage compartment from leaking out, thereby maintaining a constant temperature inside the storage compartment.

[0005] The opening of such refrigerator doors is performed by a separate operation by the user based on physical contact. However, as refrigerator functions become smarter, refrigerators capable of automatically opening the door based on the user's intent without physical contact are being proposed.

[0006] Various examples of the present disclosure may provide a refrigerator and a method for controlling the same that automatically open a door according to the user's intention by using a sensor to generate a point cloud and identifying a user's gesture based on the point cloud.

[0007] According to one example, a refrigerator comprises at least one storage compartment; at least one door configured to open and close the at least one storage compartment; at least one sensor; and at least one processor, wherein the at least one processor may be configured to acquire point cloud data using the at least one sensor, identify a gesture of a user located within a first distance from the refrigerator based on the point cloud data, and perform control to open the door of the at least one door based on the identified gesture.

[0008] According to one example, a method for controlling a refrigerator may include: acquiring point cloud data using at least one sensor; identifying a gesture of a user located within a first distance from the refrigerator based on the point cloud data; and opening the door of the refrigerator based on the identified gesture.

[0009] According to one example, a refrigerator can identify a user's gesture with high accuracy by detecting the gesture based on point cloud data acquired by a radar sensor. According to one example, the refrigerator can increase the accuracy of judging the user's gesture by learning the user's characteristics. The refrigerator can increase the convenience and efficiency of using the refrigerator by automatically opening the door according to the user's intent. The refrigerator can provide user safety by disabling the automatic door opening function when restricted user and / or animal gestures are identified.

[0010] The effects obtainable from the examples of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the examples of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the examples of the present disclosure can also be derived by those skilled in the art from the examples of the present disclosure.

[0011] FIG. 1 is a front view showing a refrigerator according to one embodiment of the present disclosure.

[0012] FIG. 2 is a perspective view showing the interior of a refrigerator according to one embodiment of the present disclosure.

[0013] FIG. 3 is a perspective view of a refrigerator with the outer door opened according to one example.

[0014] FIG. 4 is a block diagram showing the configuration of a refrigerator according to one example of the present disclosure.

[0015] FIG. 5 is a flowchart illustrating the operation of a refrigerator according to one example of the present disclosure.

[0016] Figures 6a and 6b illustrate an exemplary method for acquiring and processing point cloud data.

[0017] FIG. 7 is a diagram illustrating the process of a refrigerator according to an example of the present disclosure determining a user's gesture from point cloud data using an artificial intelligence model.

[0018] FIG. 8 is a block diagram illustrating the configuration of a radar sensor according to one example.

[0019] FIGS. 9a to 9c are exemplary drawings showing the arrangement of sensors according to one example of the present disclosure.

[0020] FIG. 10 is an exemplary diagram showing a door control module of a refrigerator according to one example of the present disclosure.

[0021] Figures 11a and 11b illustrate a method for opening the automatic door of a refrigerator according to one example.

[0022] Figures 12a and 12b illustrate a method for automatically opening the door of a refrigerator according to one example.

[0023] Figures 13a and 13b illustrate a method for opening the automatic door of a refrigerator according to one example.

[0024] FIG. 14 is a flowchart illustrating the operation of a refrigerator according to one example of the present disclosure.

[0025] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments.

[0026] In relation to the description of the drawings, similar reference numerals may be used for similar or related components.

[0027] The singular form of the noun corresponding to an item may include one or plural items, unless the relevant context clearly indicates otherwise.

[0028] In this document, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0029] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0030] Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that the component may be connected to the other component directly (e.g., via a wire), wirelessly, or through a third component.

[0031] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this document, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] When it is said that a component is "connected," "combined," "supported," or "in contact" with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0033] When it is said that a component is located "on" another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0034] The term "and / or" includes a combination of multiple related described components or any of the multiple related described components.

[0035] The functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or AI-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or AI models stored in memory. Alternatively, if the one or more processors are AI-dedicated processors, the AI-dedicated processors may be designed with a hardware structure specialized for processing a specific AI model. The processor may perform a preprocessing process to convert data applied to the AI ​​model into a form suitable for application to the AI ​​model.

[0036] Artificial intelligence models can be created through learning. Here, being created through learning means that a basic artificial intelligence model is trained using multiple learning data by a learning algorithm, thereby creating a predefined set of behavioral rules or an artificial intelligence model configured to perform a desired characteristic (or objective). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0037] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained by the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0038] The artificial intelligence models used in the examples of the present disclosure may be implemented in various embodiments depending on the manufacturer of the electronic device or the user of the electronic device, and are not limited by the above examples.

[0039] The operating principle and examples of the present invention will be described below with reference to the attached drawings.

[0040] FIG. 1 is a front view showing a refrigerator (1) according to one embodiment of the present disclosure, and FIG. 2 is a perspective view showing the interior of a refrigerator (1) according to one embodiment of the present disclosure.

[0041] The refrigerator (1) may include a main body (10), a storage room (20) provided inside the main body (10) so that the front is open, and a door (300) rotatably coupled to the main body (10) to open and close the open front of the storage room (20).

[0042] The main body (10) can form the exterior of the refrigerator (1). The main body (10) may include an inner body (11) that forms a storage room (20) and an outer body (12) that is coupled to the outer side of the inner body (11) to form the exterior. Additionally, the main body (10) may further include a cold air supply device (not shown) that supplies cold air to the storage room.

[0043] The cold air supply device may be configured to include components such as a compressor, a condenser, an expansion valve, an evaporator, a blower fan, and a cold air duct. Insulating material (not shown) may be filled between the inner surface (11) and the outer surface (12) of the main body (10) to prevent cold air from leaking out of the storage room.

[0044] A machine room (not shown) may be provided at the rear lower side of the main body (10), in which a compressor for compressing the refrigerant and a condenser for condensing the compressed refrigerant are installed.

[0045] The storage room may be divided into multiple sections by a horizontal partition (21) and a vertical partition (22). In this embodiment, the storage room may include an upper storage room (20a) and a lower storage room (20b). The storage room may be provided with a shelf (23) on which food can be placed and a sealed container (24) for storing food in a sealed container. The storage room is provided with an open front so that food can be taken out and put in, and the open front can be opened and closed by a door (300).

[0046] The upper storage room (20a) can be opened and closed by a plurality of doors (300a, 300b). The lower storage room (20b) can be opened and closed by a plurality of doors (300c, 300d).

[0047] The refrigerator (1) may further include a handle (100) provided on the door (300). The user can easily open and close the door (300) by gripping the handle (100). The handle (100) may be formed to be long along the vertical direction (Z) of the door (300).

[0048] The refrigerator (1) may further include a dispenser (not shown). The dispenser may be installed on the door (300). As an example, the dispenser may be installed on the upper left door (300a). Through the dispenser, the user can directly dispense water or ice to the outside without opening the door (300a). The dispenser may include a cavity formed by being recessed into the inner side of the door (300a) to form a dispensing space. The cavity may be provided with a dispensing port for dispensing water or ice and a dispensing lever for dispensing water or ice. When the dispensing lever is pressed, water or ice is dispensed from the dispensing port. The dispenser may further include a dispenser status display window that displays the operating status of the dispenser. The dispenser status display window may be equipped with a touch function.

[0049] In one embodiment, the refrigerator (1) may further include a display (200).

[0050] The display (200) can be installed on the door (300) for the convenience of the user. Specifically, the display (200) can be installed on the front (301) of the door (300).

[0051] In the following description, the display (200) is shown installed on the right upper door (300b), but any door (300) is sufficient for the display (200) to be installed, and it is not limited to the right upper door (300b). However, the following description focuses on the case where the display (200) is installed on the right upper door (300b).

[0052] The upper part of the display (200) can be placed in the same position as the upper part of the handle (100) in the vertical direction (Z) of the door (300).

[0053] The bottom part of the display (200) can be positioned at the same location as the bottom part of the dispenser (40) in the vertical direction (Z) of the door (300). One side end of the display (200) adjacent to the handle (100) can be spaced apart from the handle (100) at a certain distance. The other side end facing the one side end of the display (200) adjacent to the handle (100) can be spaced apart from the edge of the door (300) at a certain distance.

[0054] In another aspect, the display (200) may have a rectangular shape with a long side in the vertical direction (Z) of the door (300). The display (200) may include a right long side facing the right side of the door (300), a left long side facing the left side of the door (300), an upper short side facing the upper side of the door (300), and a lower short side facing the lower side of the door (300).

[0055] The right long side may be spaced at a certain distance from the right edge of the door (300) toward the left side of the door (300). The left long side may be spaced at a certain distance from the handle (100) toward the right side of the door (300). The upper short side may be positioned at the same location as the upper part of the handle (100) in the vertical direction (Z) of the door (300). The lower short side may be positioned at the same location as the lower part of the dispenser (40) in the vertical direction (Z) of the door (300).

[0056] By arranging the display (200) in this way, it is possible to implement a neat and stable design for the refrigerator (1).

[0057] The display (200) may include a display panel (220) and a touch panel (221). However, the display (200) may include only the display panel (220). The display (200) may be equipped with a wake-up function that is automatically activated when a user approaches within a certain range. For example, the wake-up function may be implemented through a sensor (e.g., sensor (162) of FIG. 4).

[0058] Specifically, when the sensor (162) detects the approach of a user within a certain range, the display (200) can be activated. That is, the display (200) can be turned on. Conversely, if the sensor (162) does not detect the approach of a user within a certain range, the display (200) may not be activated. That is, the display (200) may remain off. When the display (200) is activated, various videos or images can be displayed on the display (200).

[0059] For example, the display (200) may have a function to pause the video and turn off the power of the display (200) when the user opens the door equipped with the display (200). Additionally, the display (200) may have a function to resume playback of the video and turn on the power of the display (200) when the user closes the door equipped with the display (200). For example, the power off and on functions of the display (200) may be implemented through a door opening / closing sensor (e.g., the opening / closing sensor (163) of FIG. 4).

[0060] The display (200) may include a display panel (220). The display (200) may include a liquid crystal display (LCD). The display panel (220) may be located on the front of the display. A touch panel (221) may be formed on the display panel (220).

[0061] The user can play or pause a video by touching the touch panel (221). The touch panel (221) may be implemented as a capacitive or pressure-sensitive type. However, the method of forming the touch panel (221) is not limited to the above example.

[0062] The display panel (220) may be provided with at least one input UI component (User Interface Component). The at least one input UI component may include, as an example, a camera UI component that activates a camera (e.g., sensor (162) of FIG. 4), a list UI component that lists various lists related to the refrigerator (1) functions, a home UI component that returns to the start screen, a return UI component that returns to the previous step, and an information UI component that provides information about the overall functions of the refrigerator (1) or the overall functions of the display (200).

[0063] At least one input UI component may be formed on the display panel (220). Preferably, at least one input UI component may be formed on an outer area of ​​the display panel (220) so as not to interfere with an image or video displayed on the display (200).

[0064] The refrigerator (1) may further include at least one microphone for implementing a voice recognition function. A voice command input through at least one microphone is transmitted to a processor (e.g., processor (191) of FIG. 4), and the processor (191) controls a display (200) to display the result of the voice command.

[0065] The refrigerator (1) may further include an illuminance sensor. The illuminance sensor can reduce power loss of the refrigerator (1) by adjusting the brightness of the display in bright places and adjusting the brightness of the display in dark places. The detection result of the illuminance sensor is transmitted to a processor (191), and the processor (191) controls the display panel (220) to adjust the illuminance of the display (200).

[0066] The refrigerator (1) may include a door opening / closing sensor (163). The door opening / closing sensor (163) may be provided at a hinge (not shown) connecting the door (300) and the main body (10), or at a part of the door (300) or the main body (10) where the door (300) and the main body (10) come into contact.

[0067] At least one of the sensor (162), at least one microphone, and at least one light sensor may be placed on the front of the door (300). For example, at least one of the sensor (162), at least one microphone, and at least one light sensor may detect a change in the front (X) of the refrigerator (1).

[0068] At least one of the sensor (162), at least one microphone, and at least one of the light sensor may be placed on the back of the door (300). For example, at least one of the sensor (162), at least one microphone, and at least one of the light sensor may detect changes in the interior of the refrigerator (1) (e.g., storage room (20)).

[0069] FIG. 3 is a perspective view of a refrigerator (1) according to one example with the outer door (120) opened.

[0070] The refrigerator (1) may include at least one door (300). At least one of the doors (300) may be configured as a double door having an inner door (110) and an outer door (120). For example, the left upper door (300a) may include an inner door (110) and an outer door (120).

[0071] The inner door (110) can be rotatably connected to the main body (10) via a hinge. The inner door (110) may have an opening (101). The opening (101) may be formed in the central part of the inner door (110), excluding the rim portion.

[0072] At least one door basket (102; 104) may be installed in the opening (101). A portion of the door basket (102; 104) may be positioned on the rear side of the interior door (110).

[0073] An outer door (120) may be provided to open and close the opening (101) of the inner door (110). When the outer door (120) is opened, the opening (101) of the inner door (110) can be accessed. The outer door (120) may be rotatably connected to the inner door (110) via a hinge. The outer door (120) may rotate in the same direction as the inner door (110).

[0074] The outer door (120) may be opaque. However, the outer door (120) is not limited thereto. For example, the outer door (120) may be a transparent door in which part is transparent. Accordingly, the rear side of the outer door (120) can be seen without opening the outer door (120). For example, the opening (101) of the inner door (110) can be seen.

[0075] The outer door (120) may have a size corresponding to the size of the inner door (110). The outer door (120) may cover the entire area of ​​the inner door (110). However, the size of the outer door (120) is not limited thereto and may be smaller than the size of the inner door (110).

[0076] The outer door (120) may be provided with a latch (121) for securing to the inner door (110), and the inner door (110) may be provided with a catch (111) to engage with the latch (121).

[0077] When the outer door (120) is opened while the latch (121) and the catch (111) are engaged, the outer door (120) and the inner door (110) open together, and when the outer door (120) is opened while the latch (121) and the catch (111) are not engaged, only the outer door (120) opens and the inner door (110) does not open.

[0078] A decorative panel (not shown) can be detachably attached to the front of the exterior door (120).

[0079] According to one example, the door (300) of the refrigerator (1) may further include a door interior space (103) formed on the rear side of the inner door (110). The door interior space (103) may be an independent space from the storage room (20) so that food can be stored separately from the storage room (20). The door interior space (103) may have a different temperature from the storage room (20).

[0080] The refrigerator (1) may include a door control module for controlling the opening or closing of an inner door (110). The refrigerator (1) may include a door control module for controlling the opening or closing of an outer door (120).

[0081] FIG. 4 is a block diagram showing the configuration of a refrigerator (1) according to an example of the present disclosure.

[0082] Referring to FIG. 4, the refrigerator (1) may include a door control module (150), a sensor unit (160), a cooling unit (170), a communication unit (180), a control unit (190), and a display (200).

[0083] The door control module (150) can control the opening or closing of at least one door. The door control module (150) may include a motor drive unit (151) and a motor (152). For example, the door control module (150) can precisely control the movement of at least one door (300) depending on whether the at least one door (300) is open or the degree of opening.

[0084] The motor drive unit (151) can control the motor. For example, the motor drive unit (151) can activate or deactivate the motor (152). For example, the motor drive unit (151) can control the operating state of the motor (152) by supplying or cutting off power to the motor (152).

[0085] The motor (152) can open at least one door by rotating. For example, the motor (152) can open at least one door by being activated based on the control of the motor drive unit. For example, the motor (152) can close at least one door by being deactivated based on the control of the motor drive unit.

[0086] The sensor unit (160) may include a temperature sensor (161), a sensor (162), and a door opening / closing sensor (163).

[0087] The temperature sensor (161) can detect the temperature around the refrigerator (1) or inside the refrigerator (1). For example, the temperature sensor (161) may include a plurality of temperature sensors that detect the temperature inside the storage room (20). For example, the temperature sensor (161) may include a plurality of temperature sensors that detect the external temperature around the refrigerator (1).

[0088] For example, a plurality of temperature sensors may be installed in each of the plurality of storage rooms (20) to detect the temperature of each of the plurality of storage rooms (20) and output an electrical signal corresponding to the detected temperature to the control unit (190). Each of the plurality of temperature sensors may include a thermistor whose electrical resistance changes according to temperature.

[0089] The sensor (162) is configured to sense various information, for example, point cloud data. For example, the sensor (162) may include at least one of a radar (RaDAR, Radio Detection And Ranging) sensor, an optical sensor, or an ultrasonic sensor. As an example, the sensor (162) may be a radar sensor.

[0090] Radar sensors transmit electromagnetic waves (radio waves) and can generate point clouds by measuring the time it takes for the waves to reflect off an object and return. The wavelengths of the electromagnetic waves used by radar sensors mainly belong to the microwave or millimeter wave bands. For example, microwave radar can use wavelengths from 10 mm to 300 mm. For example, millimeter wave radar can use wavelengths from 1 mm to 10 mm. Longer wavelengths may be advantageous for long-range detection (e.g., microwave band). Shorter wavelengths may enable more precise detection (e.g., detection of small objects) (e.g., millimeter wave band). For example, the sensing distance using a radar sensor may be within 8 m or between 1 and 3 m. Radar sensors may have a field of view (FoV). The field of view of a radar sensor may include a horizontal FoV and a vertical FoV. The horizontal field of view is the range in which the radar sensor can sense in the horizontal direction. For example, the horizontal field of view may be 60 degrees. The vertical field of view is the range in which the radar sensor can sense in the vertical direction. For example, the vertical field of view may be 40 degrees. The radar sensor can detect objects within a certain distance from the radar sensor. The configuration of the radar sensor will be described later with reference to FIG. 8.

[0091] The optical sensor may include at least one of a LiDAR (Light Detection And Ranging) sensor, a structured light sensor, a laser scanner, or a camera sensor. A LiDAR sensor can generate a point cloud by transmitting high-energy laser pulses and measuring the time it takes for the laser pulses to reflect back from an object. A structured light sensor can generate a point cloud by projecting a pattern onto an object using a camera and a projector and analyzing how the pattern is distorted on the object's surface. A laser scanner can generate a point cloud by scanning the surface of an object using a laser beam and measuring the distance through the reflected light. A camera sensor may be a sensor that generates a digital image by converting light collected in a sensing area into an electrical signal. For example, a camera sensor may include at least one of a Time of Flight (ToF) sensor, a stereo camera, or an RGB-D (RGB-Depth) camera. A ToF sensor can transmit light to an object and measure the time it takes for the reflected light to return. An RGB-D camera is a device that combines an RGB camera and a depth sensor, capable of capturing 2D images while simultaneously measuring depth information. A stereo camera may include two or more cameras. A stereo camera can estimate the depth of each pixel using the disparity between images acquired using two or more cameras, thereby generating a 3D point cloud.

[0092] An ultrasonic sensor can generate a point cloud by transmitting ultrasound and measuring the time it takes for the ultrasound to reflect off an object and return.

[0093] As described above, the door opening / closing sensor (163) can output whether the door (300) is open or closed as a preset judgment value. For example, the door opening / closing sensor (163) can output 1 if the door (300) is open and output 0 if the door (300) is closed.

[0094] The door opening / closing sensor (163) can also be implemented as a distance sensor, and if the distance between the door (300) and the main body is greater than or equal to a reference distance, it can be determined that the door (300) is open, and if it is less than the reference distance, it can be determined that the door (300) is closed. However, the door opening / closing sensor (163) is not limited to this, and there are no restrictions on its configuration as long as it can determine whether the door (300) is open and output a preset judgment value.

[0095] The cooling unit (170) can supply cooled air to the storage room. Specifically, the cooling unit (170) can maintain the temperature of the storage room within a range specified by the user by utilizing the circulation of refrigerant in the refrigerant circuit.

[0096] The cooling unit (170) may include a compressor (171) that compresses the refrigerant in a gaseous state, a condenser (172) that converts the compressed gaseous refrigerant into a liquid state, an expander (173) that reduces the pressure of the liquid refrigerant, and an evaporator (174) that converts the reduced pressure liquid refrigerant into a gaseous state. The cooling unit (170) can cool the air in the storage room by utilizing the phenomenon in which the liquid refrigerant absorbs thermal energy from the surrounding air while converting into a gaseous state.

[0097] However, the cooling unit (170) is not limited to including a refrigerant circuit. For example, the cooling unit (170) may include a Peltier element utilizing the Peltier effect or a magnetic cooling material utilizing the magneto-caloric effect.

[0098] The communication unit (180) can exchange data with external devices such as a server device and / or a user device and / or a cooking device.

[0099] The communication unit (180) may include a wired communication module (182) that exchanges data with external devices via a wire, and a wireless communication module (181) that exchanges data with external devices wirelessly.

[0100] The wired communication module (182) can connect to a wired communication network and communicate with external devices through the wired communication network. For example, the wired communication module (182) can connect to a wired communication network via Ethernet (Ethernet, IEEE 802.3 technical standard) and receive data from external devices through the wired communication network.

[0101] The wireless communication module (181) can communicate wirelessly with a base station or an access point (AP) and can connect to a wired communication network through the base station or access point. The wireless communication module (181) can also communicate with external devices connected to the wired communication network via the base station or access point. For example, the wireless communication module (181) can communicate wirelessly with the access point (AP) using Wi-Fi (IEEE 802.11 technical standard) or communicate with the base station using CDMA, WCDMA, GSM, LTE (Long Term Evolution), or WiBro. The wireless communication module (181) can also receive data from external devices via the base station or access point. Additionally, the wireless communication module (181) can communicate directly with external devices. For example, the wireless communication module (181) can receive data wirelessly from external devices using Wi-Fi, Bluetooth (Bluetooth, IEEE 802.15.1 technical standard), or ZigBee (ZigBee, IEEE 802.15.4 technical standard).

[0102] In this way, the communication unit (180) can transmit or receive data with external devices, and in particular, receive video data including video and / or audio from external devices and output the received data to the control unit (190).

[0103] The control unit (190) processes user input and / or door opening / closing detection data and / or communication data, and can control the components included in the refrigerator (1) based on the data processing.

[0104] The control unit (190) includes a memory (192) for storing / remembering programs and / or data, and a processor (191) for processing user input and / or door opening / closing detection data and / or communication data according to the programs and / or data stored in the memory (192).

[0105] The memory (192) can store / remember programs and / or data. A program may include a plurality of instructions combined to perform a specific function, and data may be processed and / or manipulated by a plurality of instructions included in the program. Additionally, the programs and / or data may include system programs and / or system data directly related to the operation of the refrigerator (1), and application programs and / or application data that provide convenience to the user.

[0106] The memory (192) may include a non-volatile memory for storing programs and / or data for controlling components included in the refrigerator (1), and a volatile memory for storing temporary data that occurs while controlling components included in the refrigerator (1).

[0107] Non-volatile memory can store programs and / or data electrically, magnetically, or optically, for example. Non-volatile memory may include, for example, ROM (Read Only Memory) and flash memory for storing data for a long period. Additionally, non-volatile memory may include solid disk drives (SSDs), hard disk drives (HDDs), or optical disk drives (ODDs).

[0108] Volatile memory can, for example, load programs and / or data from non-volatile memory and electrically store programs and / or data. Volatile memory may include, for example, S-RAM (Static Random Access Memory) and / or D-RAM (Dynamic Random Access Memory) for temporarily storing data.

[0109] This memory (192) can store / remember programs and data such as an operating system (OS), middleware, and applications, and can provide programs and data to the processor (191) in response to a request from the processor (191).

[0110] The processor (191) can process user input of the display (200) and / or detection data of the sensor (162) and / or communication data of the communication unit (180) according to the program and / or data stored in the memory (192). Additionally, the processor (191) can generate a control signal to control the operation of the display (200) and / or the communication unit (180) based on the data processing.

[0111] FIG. 5 is a flowchart showing the operation of a refrigerator (1) according to an example of the present disclosure.

[0112] In the following examples of operations, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of the operations may be changed, or at least two operations may be performed in parallel. At least some of the operations may be performed on a server.

[0113] Referring to FIG. 5, the refrigerator (1) of the present disclosure can automatically open the door by using at least one sensor to acquire point cloud data corresponding to a user and identifying a user's gesture instructing the door to open from the point cloud data.

[0114] According to one embodiment, the refrigerator (1) can acquire point cloud data using at least one sensor (e.g., sensor (162) of FIG. 4) (operation 510), identify a gesture of a user located within a first distance from the refrigerator based on the point cloud data (operation 520), and control a door control module (e.g., door control module (150) of FIG. 4) to open a first door based on the identified gesture being identified as a first gesture (operation 530).

[0115] According to one embodiment, the refrigerator (1) can acquire point cloud data using a sensor (162) in operation 510. The sensor (162) may include various sensors capable of acquiring point cloud data. The point cloud data may include a point cloud contained in a frame captured by the sensor (162) at a specific time. For example, the sensor (162) performs tens or hundreds of scans per second, and the scanned information may be acquired as a frame at a specific time. The frame may include a point cloud corresponding to an object (e.g., a person, an animal) included in the sensing range of the sensor (162). The point cloud may be a set of individual points having coordinate values ​​in 3D. Each of the multiple points included in the point cloud may include coordinate values ​​of the x-axis, y-axis, and z-axis. The refrigerator (1) may acquire multiple points corresponding to an object included in the sensing range (or field of view) of the sensor (162). Multiple points included in the point cloud correspond to objects included within the sensing range of the sensor (162). The refrigerator (1) can acquire multiple frames containing the point cloud. Each frame may include the point cloud along with time information. The refrigerator (1) can acquire point cloud data based on frames acquired by the sensor (162) during a certain time interval. The point cloud data may include the point cloud contained in the frames captured by the sensor (162) during a certain time interval.

[0116] According to one embodiment, the refrigerator (1) can identify a user's gesture located within a first distance from the refrigerator based on point cloud data in operation 520. The first distance may be determined according to the sensing range of the sensor (162). The first distance may be the distance between an object included in the sensing range of the sensor (162) and the refrigerator. The refrigerator (1) can identify a user's gesture included in the sensing range of the sensor (162). The refrigerator (1) can identify a user's gesture located within the field of view (FoV) of the sensor (162). The refrigerator (1) can identify a user's gesture located in the front direction of the refrigerator (1) (the direction looking from the door of the refrigerator (1) toward the outside of the refrigerator (1)). The refrigerator (1) can identify a user's gesture after identifying that the user has been located within the first distance for a specified amount of time based on point cloud data. The refrigerator (1) can cluster a plurality of points included in the point cloud. The refrigerator (1) can analyze and classify objects based on clustered point clouds. The refrigerator (1) can identify objects (e.g., people) through clustering. For example, the refrigerator (1) can identify people. Once a person is identified, the refrigerator (1) can cluster multiple points corresponding to the person's major body joints and extract a skeleton based on this. The refrigerator (1) can track the movement of the skeleton. The movement of the skeleton may include not only the movement of the skeleton during a specific time interval (e.g., changes in the position of major body joints), but also specific postures (poses) and pose transformations taken by the skeleton. The refrigerator (1) can identify a person's gestures based on the tracked movement of the skeleton. The refrigerator (1) can identify a person's gestures using an artificial intelligence model.For example, the refrigerator (1) can infer an intended gesture from point cloud data using an artificial intelligence model trained to identify user gestures. The intended gesture may be a gesture implemented by the user through body movements or body poses. For example, the intended gesture may be a gesture actually performed by the user to provide a preset gesture (e.g., a first gesture) to the refrigerator (1). The refrigerator (1) can infer the gesture intended to be provided by the user (e.g., a first gesture) based on the point cloud data.

[0117] According to one embodiment, the refrigerator (1) may control a door control module to open a first door based on the fact that, in operation 530, the identified gesture is identified as a first gesture. The first door may be selected from at least one door included in the refrigerator (1). The first door may be one or more doors. The refrigerator (1) may include at least one door (e.g., one, two, three, four), and the first door may be one or more of the at least one door. The first gesture may be a preset gesture to open the first door. The first gesture may include a pose or movement. For example, the first gesture may be a finger spread pose in which the user opens the palm and spreads each finger apart from each other. For example, the first gesture may be a gesture in which the user raises their hand from bottom to top by more than 50 cm. The refrigerator (1) may add or change the first gesture according to the user's input. For example, the refrigerator (1) may add a gesture of shaking the head left and right, a gesture of clapping, or a gesture of moving the feet according to the user's input. The refrigerator (1) can recognize a command to open the first door from the first gesture. For example, the first door may be a pre-set door. The refrigerator (1) can set the first door to a specific door. For example, the refrigerator (1) can set the first door to an upper door. For example, the refrigerator (1) can set the first door to a lower door. The refrigerator (1) may be designed so that only the pre-set first door (e.g., upper door) opens automatically in response to the first gesture. For example, the refrigerator (1) can select the first door to open in response to the first gesture according to user input. For example, the refrigerator (1) can map the first door to open in response to the first gesture to a specific door according to user input. In response to the first gesture, the refrigerator (1) can open the first door mapped to the first gesture.The refrigerator (1) may provide a user interface that allows the user to select a first door to open in response to a first gesture. For example, the user interface may be provided through a display of the refrigerator (1) (e.g., the display (200) of FIG. 4). For example, the user interface may be provided through a server and / or device connected to the refrigerator (1). The refrigerator (1) may control a door control module to open the first door. For example, the door control module (150) may automatically open the first door using the rotational force of a motor.

[0118] According to one embodiment, a refrigerator (1) may identify a second gesture of a user located within a first distance from the refrigerator (1) based on cloud point data acquired using at least one sensor, and may perform control to open a second door based on the identified second gesture. The second gesture may be different from the first gesture, and the second door may be different from the first door. The second door may be one or more of at least one door. FIGS. 6a and 6b illustrate an exemplary method for acquiring and processing point cloud data.

[0119] In FIG. 6a, the refrigerator (1) can acquire (or generate) point cloud data based on sensing data acquired by the sensor (162). The refrigerator (1) can acquire point cloud data by processing the sensing data. For example, the refrigerator (1) can remove noise from the sensing data and / or filter it. Based on the sensing data, the refrigerator (1) can extract information such as the distance and / or angle between the object in the scene and the sensor. The refrigerator (1) can convert the extracted information into point cloud data containing points (610) in a three-dimensional coordinate system.

[0120] The sensor (162) may include a sensor that measures the depth value of an object. For example, the sensor (162) may include at least one of a radar sensor, a light sensor, or an ultrasonic sensor. For example, the light sensor may include at least one of a LiDAR sensor, an RGB-D sensor, a depth sensor, or a ToF sensor. The sensor (162) may acquire sensing data including a depth value. The refrigerator (1) may acquire point cloud data based on the sensing data.

[0121] According to one example, the sensor (162) may include a radar sensor. The sensor (162) may transmit electromagnetic waves to an object and receive a reflected signal to obtain at least one of the distance, bearing, or speed to the object. The sensor (162) may obtain 3D position information of the object by combining at least one of the distance, bearing, or speed to the object.

[0122] According to one embodiment, the sensor (162) may include an image sensor (e.g., an RGB sensor). The sensor (162) may acquire sensing data (e.g., a color image) corresponding to an object. The refrigerator (1) may estimate the depth of a scene (or object) from 2D sensing data in order to acquire point cloud data based on the sensing data. For example, the refrigerator (1) may estimate the depth of the scene using a vSLAM (visual simultaneous localization and mapping) algorithm. The refrigerator (1) may acquire point cloud data using the sensing data and the estimated depth.

[0123] According to one embodiment, the sensor (162) may include a stereo camera (e.g., two image sensors). For example, two image sensors may be placed in the refrigerator (1) at regular intervals. Each of the two image sensors may capture an object at the same time. Each of the two image sensors may acquire a color image corresponding to the object. The refrigerator (1) may estimate the depth of the scene from the two color images. The refrigerator (1) may generate a point cloud using the sensing data and the estimated depth.

[0124] According to one embodiment, the sensor (162) may be composed of a combination of two or more types of sensors. The refrigerator (1) can time-synchronize the sensing data obtained by two or more types of sensors. In one example, the sensor (162) may be composed of an image sensor and a ToF sensor. The refrigerator (1) can generate a point cloud using a color image obtained by the image sensor and a depth value obtained by the ToF sensor. In one example, the sensor (162) may be composed of an RGB-D sensor and a LiDAR sensor. The refrigerator (1) can generate a point cloud based on the sensing data obtained by each of the RGB-D sensor and the LiDAR sensor. According to one example, by using more types of sensors, a point cloud with high accuracy can be generated.

[0125] According to one embodiment, when the sensing data consists of a plurality of still images (e.g., frames), the refrigerator (1) can align point clouds corresponding to various positions and angles by using a plurality of frames of the plurality of still images.

[0126] In FIG. 6b, the refrigerator (1) can process point cloud data. For example, the refrigerator (1) can perform single frame processing. For example, the refrigerator (1) can cluster point clouds contained in a single frame acquired at a specific time. The refrigerator (1) can acquire point cloud data and, through clustering, identify information such as the location, size, or shape of at least one object in space. Each cluster can correspond to an object (e.g., person, animal) within the point cloud. For example, the refrigerator (1) can cluster spatially dense points using the DBSCAN (Density-Based Spatial Clustering of Application with Noise) algorithm. The refrigerator (1) can include points with similar movement trends (speed and / or direction) in the cluster and remove points with dissimilar movement trends (e.g., noise) from the cluster. For example, the refrigerator (1) can cluster point clouds using an artificial intelligence model (e.g., DeepCluster). For example, as illustrated in FIG. 6b, the refrigerator (1) can detect multiple objects by clustering a point cloud. For example, the point cloud may include a first cluster (620) and a second cluster (630). The refrigerator (1) can generate a three-dimensional bounding box (625; 635) corresponding to each of the detected multiple clusters (620; 630). The three-dimensional bounding box (625; 635) is in the shape of a rectangular prism and may include information such as a center point (x, y, z) or dimensions (width, height, depth). Hereinafter, "three-dimensional bounding box" may be referred to as "bounding box". According to one embodiment, the refrigerator (1) can manually generate the bounding box (625; 635) using a labeling tool.According to one embodiment, the refrigerator (1) can automatically generate bounding boxes (625; 635) using a clustering algorithm. The bounding boxes (625; 635) can define an area where an object exists and provide the location and size of the object. The bounding boxes (625; 635) can be used for at least one of object detection, object classification, tracking, or skeleton analysis. The refrigerator (1) can detect a specific object (e.g., person, animal) using the bounding boxes (625; 635). The refrigerator (1) can classify an object inside the bounding boxes (625; 635) into a specific class (e.g., cat, dog). The refrigerator (1) can track a movement path by continuously detecting a specific object in frames using the bounding boxes (625; 635). The refrigerator (1) can detect the approximate location of an object (e.g., a person) using a bounding box (625; 635) and then perform skeleton extraction in detail.

[0127] For example, the refrigerator (1) can perform sequential frames processing. The refrigerator (1) can track clusters based on multiple frames. The refrigerator (1) can identify whether a cluster corresponds to a moving object. The refrigerator (1) can measure the median speed of the cluster. The median speed of the cluster may be the median value of the speeds of all points belonging to the cluster. The refrigerator (1) can identify a cluster as a moving object if the median speed exceeds a threshold. The refrigerator (1) can identify a cluster as a stationary object if the median speed is below the threshold. The refrigerator (1) can track each of multiple clusters. The refrigerator (1) can distinguish between moving objects and stationary objects and track each cluster corresponding to the moving object and the stationary object.

[0128] FIG. 7 is a diagram showing the process of a refrigerator according to an example of the present disclosure determining a user's gesture (710) from point cloud data using an artificial intelligence model (700).

[0129] Referring to FIG. 7, the refrigerator (1) can determine the user's intention (e.g., intention to open the door or intention to close the door) from point cloud data using an artificial intelligence model (700). For example, the refrigerator (1) can determine the user's intention by inferring the user's intended gesture (710) from the point cloud data using an artificial intelligence model (700) that has learned the correlation between the point cloud data and a pre-set gesture.

[0130] The intended gesture (710) may be a gesture implemented by the user through body movements or body poses. For example, the intended gesture (710) may be a gesture actually performed by the user to provide a preset gesture (e.g., a first gesture (720)) to the refrigerator (1). The refrigerator (1) may infer the intended gesture (710) from point cloud data and determine the gesture intended to be provided by the user (e.g., a first gesture (720)) from the intended gesture (710).

[0131] The artificial intelligence model (700) used by the refrigerator (1) for inferring the intention gesture (710) may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. The artificial intelligence model (700) may be composed of a neural network (or artificial neural network) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network may refer to a model in which artificial neurons (nodes) forming a network through synaptic connections change the strength of the synaptic connections through learning to possess problem-solving capabilities. The neurons of a neural network may include a combination of weights or biases. A neural network may include one or more layers composed of one or more neurons or nodes. For example, a neural network may include an input layer, a hidden layer, and an output layer. A neural network can infer a result (output) to be predicted from an arbitrary input by changing the weights of the neurons through learning.

[0132] At least one processor included in the refrigerator (1) can generate a neural network, train (or learn) the neural network, perform operations based on received input data, generate an information signal based on the results of the operation, or retrain the neural network.For example, neural networks include Convolutional Neural Network (CNN), RecuREnt Neural Network (RNN), perceptron, multilayer perceptron, Feed Forward (FF), Radial Basis Network (RBF), Deep Feed Forward (DFF), Long Short Term Memory (LSTM), Gated RecuREnt Unit (GRU), Auto Encoder (AE), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics) Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning It may include, but is not limited to, a Neural Machine (1), ESN (Echo State Network), DRN (Deep Residual Network), DNC (Differentiable Neural Computer), NTM (Neural Turning Machine), CN (Capsule Network), KN (Kohonen Network), and AN (Attention Network). At least one processor included in the refrigerator (1) may include one or more processors for performing operations according to the models of the neural network.

[0133] According to one embodiment, the refrigerator (1) can identify an object (e.g., a person) from point cloud data and track the movement of the identified object. The refrigerator (1) can identify a gesture based on the movement of the tracked object. The refrigerator (1) can determine whether the identified gesture is a first gesture (720).

[0134] According to one embodiment, the refrigerator (1) can recognize objects belonging to a space based on the analysis and classification of objects. The refrigerator (1) may use an artificial intelligence model (700) to recognize objects. The artificial intelligence model (700) may include an object recognition model. The object recognition model can analyze and classify objects by taking a clustered point cloud as input. The object recognition model can output probability values ​​that an object within the point cloud data belongs to each class corresponding to the category of the object. Through the output value of the object recognition model, it can be determined which class an object belongs to. The object recognition model can extract features within the point cloud data and classify the class of an object based on the extracted features. For example, the object recognition model can classify the class of an object by obtaining the location and feature value where an object is presumed to exist within an image, and by performing calculations on the obtained feature value along the nodes and edges included in each layer of fully connected layers. The object recognition model can output a calculated probability value for each class using an activation function such as a softmax function.

[0135] An object recognition model can output the top n probability values ​​(or scores) among all class-specific probability values ​​of objects that the model can recognize as an object recognition result. The higher the output value, the higher the probability that the object belongs to that class. However, the output value must be greater than a predetermined threshold, which is the criterion for determining whether the object recognition result is reliable. For example, as an object recognition result, the object recognition model can output '0.35', '0.15', '0.13', '0.11', and '0.1' as probability values ​​for the top 5 classes. In this case, if the predetermined threshold, which is the criterion for determining whether the object recognition result is reliable, is '0.4', then since none of the output probability values ​​are greater than '0.4', the object recognition through the object recognition model may be treated as a failure.

[0136] According to one embodiment, the object recognition model may include a person recognition model. The refrigerator (1) can recognize a person using the person recognition model within the frame.

[0137] According to one embodiment, the refrigerator (1) can collect user gesture data using a sensor (e.g., sensor (162) of FIG. 4). The gesture data may be point cloud data. The refrigerator (1) can extract gesture features from the point cloud data and identify the gesture based on the extracted gesture features.

[0138] According to one embodiment, the refrigerator (1) can extract features of a gesture using skeleton analysis and identify the gesture based on the extracted features of the gesture. Skeleton analysis is a process of extracting a human body structure and analyzing it to identify a gesture. The refrigerator (1) can extract the positions of major body joints of a human (e.g., head, elbows, knees) and identify them as skeleton points. For example, the refrigerator (1) can extract the positions of major body joints using a skeleton tracking model or a pose estimation model (e.g., OpenPose, HRNet).

[0139] The refrigerator (1) can track the movement of the skeleton. For example, the refrigerator (1) can track the position change and velocity of the skeleton point. The refrigerator (1) can identify human gestures based on the tracked movement of the skeleton. For example, the refrigerator (1) can identify human gestures based on the position change and velocity of the tracked skeleton point. The refrigerator (1) can track the movement of the skeleton based on multiple frames and identify gestures based on the tracked movement of the skeleton. The refrigerator (1) can extract features of the movement of the skeleton. For example, the refrigerator (1) can use the relative distance or angle between specific joints (e.g., distance between the wrist and shoulder, angle between the elbow and knee) as features of the shape of the skeleton. For example, the refrigerator (1) can extract the speed and acceleration of the joint position changing over time as features of the movement of the skeleton. The refrigerator (1) can label gestures so that the artificial intelligence model (700) can learn the skeleton shape or movement pattern associated with a specific gesture. For example, the refrigerator (1) can assign labels such as "arm-waving gesture," "hand-raising gesture," and "foot-moving gesture" in response to the gesture.

[0140] The refrigerator (1) can learn the characteristics of a skeleton. Based on the characteristics of the skeleton, the refrigerator (1) can identify the user's gestures from point cloud data. For example, the refrigerator (1) can store the skeleton data identified from the point cloud data in memory (192). The skeleton data may include the locations of the joints included in the skeleton, or the relationships between the joints. As the skeleton data is accumulated and stored in memory (192), a database of the user's skeleton can be created. The refrigerator (1) can analyze the locations of the joints and / or the relationships between the joints from the database of the user's skeleton stored in memory (192). The refrigerator (1) can learn the unique characteristics of the user's skeleton that distinguish it from other users' skeletons by using at least one artificial intelligence model (700). The refrigerator (1) can quickly and accurately identify the user's gesture from the sensing data obtained by the sensor (162) based on the characteristics of the learned skeleton.

[0141] The refrigerator (1) can learn specific gestures using an artificial intelligence model (700). The refrigerator (1) can collect gesture data using a sensor (e.g., sensor (162) of FIG. 4). The gesture data may be point cloud data. The refrigerator (1) can identify and classify specific gestures based on the gesture data using an artificial intelligence model (700). The artificial intelligence model (700) can learn spatial and temporal characteristics of the gesture. The artificial intelligence model (700) can identify patterns related to the gesture through input data. For example, the artificial intelligence model (700) can receive feature data such as the coordinates, distance, angle, or speed of a skeleton point as input and learn a gesture class (e.g., waving, clapping) as an output label. For example, the artificial intelligence model (700) may include an RNN / LSTM that processes time-series data to learn the temporal continuity of joint movements, a Graph Convolutional Network (GCN) that represents the skeleton as a graph to learn the relationships between joints, and a 3D CNN that processes the temporal-spatial movements of joints in video data. The artificial intelligence model (700) can improve the gesture recognition rate through an iterative learning process.

[0142] The refrigerator (1) can identify gestures quickly and accurately using a trained artificial intelligence model (700). The artificial intelligence model (700) can receive real-time sensing data from a sensor. The real-time data can be preprocessed (e.g., noise removal). The artificial intelligence model (700) can output a label corresponding to the gesture in real time.

[0143] The refrigerator (1) can determine user characteristics from point cloud data obtained based on the sensor (162). For example, the refrigerator (1) can determine user characteristics by analyzing the user's skeleton characteristics (e.g., positions of joints, relationships between joints). For example, user characteristics may include at least one of the user's gender or age.

[0144] The refrigerator (1) can identify restricted users based on user characteristics. For example, restricted users may include child users and / or elderly users. Based on user characteristics, the refrigerator (1) can disable the automatic door opening function if it is determined to be a restricted user. For example, the refrigerator (1) can disable the automatic door opening function if it identifies an animal.

[0145] According to one embodiment, the refrigerator (1) can identify a user's gesture with high accuracy by detecting the user's gesture based on point cloud data. According to one embodiment, the refrigerator (1) can increase the accuracy of judging the user's gesture by learning the user's characteristics. The refrigerator (1) can increase the convenience and efficiency of using the refrigerator (1) by automatically opening the door according to the user's intention. The refrigerator (1) can identify restricted users and / or animals based on point cloud data. The refrigerator (1) can provide safety of use by disabling the automatic door opening function when the gesture of a restricted user and / or animal is identified.

[0146] According to one embodiment, the refrigerator (1) can distinguish users based on high-resolution point cloud data. The refrigerator (1) can perform customized actions (functions) designated for each user (e.g., display / interface personalization, food / ingredient recommendations tailored to preferences, etc.).

[0147] According to one embodiment, the refrigerator (1) can obtain skeleton data from point cloud data using an artificial intelligence model (700). For example, the skeleton data may include at least one of 3D joint coordinates, a skeleton shape, or a movement pattern of the skeleton. The refrigerator (1) can infer an intentional gesture based on the skeleton data using the artificial intelligence model (700). The intentional gesture may be a gesture actually performed by a user to provide the refrigerator (1) with a preset gesture (e.g., a first gesture). For example, the refrigerator (1) may determine that the intentional gesture is the first gesture if the intentional gesture shows a degree of agreement with the preset first gesture above a threshold.

[0148] The refrigerator (1) can open the first door based on the user's gesture being determined to be a first gesture for opening the door. For example, the door control module (150) can automatically open the selected door using the rotational force of a motor. The refrigerator (1) can provide a visual or auditory notification indicating that the door has been opened.

[0149] FIG. 8 is a block diagram illustrating the configuration of a radar sensor according to one example.

[0150] Referring to FIG. 8, a radar sensor (e.g., sensor (162) of FIG. 4) may include a transmitter (810), a receiver (820), and a signal processing unit (830). The radar sensor may transmit electromagnetic waves to an object and receive a reflected signal to obtain at least one of the distance, bearing, or speed to the object. The radar sensor may obtain 3D position information of the object by combining at least one of the distance, bearing, or speed to the object.

[0151] A transmitter (810) can transmit an electromagnetic signal through at least one transmitting antenna (811). Hereinafter, the signal transmitted by the transmitter (810) may be referred to as the transmission signal. The electromagnetic signal may be reflected at interfaces between the materials of objects (e.g., people, animals, metals). A receiver (820) can receive the reflected signal through at least one receiving antenna (821). Hereinafter, the signal received by the receiver (820) may be referred to as the reception signal.

[0152] The signal processing unit (830) can process radar signals and output the processing result to a processor (e.g., the processor (191) of FIG. 4). The signal processing unit (830) may include at least one of a mixer (831), a low-pass filter (832), an analog-to-digital converter (ADC) (833), or a fast-time fourier (FTF) signal processing unit (834). The signal processing unit (830) can estimate the distance to an object by analyzing radar data sensed through a radar sensor. The signal processing unit (830) can estimate the speed of an object by analyzing radar data sensed through a radar sensor. The signal processing unit (830) can estimate the bearing of an object based on the received signals corresponding to each receiving antenna of the radar sensor. The signal processing unit (830) can obtain the position (x,y,z coordinates) of an object in three-dimensional space by combining at least one of the distance, orientation, or speed to the object.

[0153] The mixing unit (831) can mix the radar transmission signal and the radar reflection signal to generate an Intermediate Frequency (IF) signal based on the difference between the transmitted signal and the received signal. The mixing unit (831) can generate the frequency difference between the transmitted signal and the received signal as the IF signal. The low-pass filter (832) can filter the low-frequency band signal among the IF signals to reduce noise of high-frequency components included in the IF signal. The analog-to-digital converter (833) can convert the IF signal, on which low-pass filtering has been performed, into a digital signal.

[0154] The FTF signal processing unit (834) may include at least one of a Range Fast Fourier Transform (FFT) signal processing unit, a Doppler Shift FFT signal processing unit, a Constant False Alarm Rate (CFAR) signal processing unit, and an Angle of Arrival FFT signal processing unit. The Range FFT signal processing unit can estimate the distance to an object. The Range FFT signal processing unit can convert a signal collected in the time domain into the frequency domain to extract frequency information along the time axis. The frequency difference between the received signal and the transmitted signal may be proportional to the distance to the object. The Range FFT signal processing unit can estimate the distance to the object based on the frequency difference between the received signal and the transmitted signal. The Doppler Shift FFT signal processing unit can estimate the velocity of the object using the frequency change while the transmitted signal is sent and the signal reflected from the object returns. The CFAR signal processing unit can filter out noise included in the received signal (e.g., signals occurring in the background other than the object). The arrival angle FFT signal processing unit can estimate the bearing of an object (in which direction the object is located relative to the radar sensors) using the angle at which the received signal reaches multiple radar sensors.

[0155] The refrigerator (1) (e.g., processor (191)) can recognize objects based on radar signals received through the signal processing unit (830) or the processing results thereof. The radar sensor can provide high-resolution 3D data (point cloud data). The refrigerator (1) can acquire very precise 3D data using the radar sensor. For example, the refrigerator (1) can identify objects separated by more than 10 cm as different objects using the radar sensor. The refrigerator (1) can identify the user's gesture with high accuracy by detecting the user's gesture based on the point cloud data acquired by the radar sensor.

[0156] FIGS. 9a to 9c are exemplary drawings showing the arrangement of a sensor (162) according to one example of the present disclosure.

[0157] In FIGS. 9a to 9c, the sensor (162) can be placed at various locations on the refrigerator (1). The sensor (162) can be placed at various locations depending on the type of refrigerator (1). The refrigerator (1) can be classified according to the shape of the storage compartment and the door. For example, the refrigerator may be an FDR (French Door Refrigerator) type refrigerator (1a) (see FIG. 9a), a TMF (Top Mounted Freezer) type refrigerator or a BMF (Bottom Mounted Freezer) type refrigerator (1b) (see FIG. 9b), or an SBS (Side by Side) type refrigerator (1c) (see FIG. 9c). In the FDR type refrigerator (1a), the storage compartment is divided vertically by a horizontal partition, with a refrigerator compartment formed on the upper side and a freezer compartment formed on the lower side, and the upper refrigerator compartment can be opened and closed by a pair of doors. In a TMF-type refrigerator, the storage compartment may be divided vertically by a horizontal partition, with a freezer compartment formed on the upper side and a refrigerator compartment formed on the lower side. In a BMF-type refrigerator (1b), the storage compartment may be divided vertically by a horizontal partition, with a refrigerator compartment formed on the upper side and a freezer compartment formed on the lower side. The TMF-type refrigerator and the BMF-type refrigerator are similar in form in that the storage compartment is divided vertically by a horizontal partition, but differ in that the freezer compartment is located on the upper side in the TMF-type refrigerator and the freezer compartment is located on the lower side in the BMF-type refrigerator. In an SBS-type refrigerator (1c), the storage compartment may be divided horizontally by a vertical partition, with a freezer compartment formed on one side and a refrigerator compartment formed on the other side. According to one embodiment, the refrigerator (1) may include a rotating door that rotates relative to the side of the refrigerator (1). According to one embodiment, the refrigerator (1) may include a drawer-type door that is pulled out to the front of the refrigerator (1). For example, the refrigerator (1b) may include at least one rotary door (300e) that opens and closes a storage compartment located at the top of the refrigerator (1b).For example, the refrigerator (1b) may include at least one drawer-type door (300f-1; 300f-2) for opening and closing a storage compartment located at the bottom of the refrigerator (1b). The refrigerator (1b) may open and close a rotary door (300e) and / or a drawer-type door (300f-1; 300f-2) using a door control module included in the refrigerator (1b).

[0158] As illustrated in FIG. 9a, the sensor (162) may be positioned between the upper door (300a; 300b) and the lower door (300c; 300d). For example, the sensor (162) may be positioned in the frame between the upper door (300a; 300b) and the lower door (300c; 300d). The sensor (162) may be formed in the center between the upper door (300a, 300b) and the lower door (300c, 300d) of the refrigerator (1a). The sensor (162) may be positioned below the upper door (300a; 300b). The sensor (162) may be positioned above the lower door (300c; 300d).

[0159] As illustrated in FIG. 9b, the sensor (162) may be positioned between the upper door (300e) and the lower door (300f-1). The sensor (162) may be positioned in the frame between the upper door (300e) and the lower door (300f-1). The sensor (162) may be positioned below the upper door (300e). The sensor (162) may be positioned above the lower door (300f-1).

[0160] As shown in FIG. 9c, the sensor (162) may be placed on the upper side of the refrigerator (1c). The sensor (162) may be placed in the frame between the left door (300g) and the right door (300h).

[0161] The placement of the sensor (162) included in the refrigerator (1) according to one example of the present disclosure is not limited to the positions shown in FIGS. 9a to 9c. Although only one sensor (162) is shown in FIGS. 9a to 9c, additional sensors (162) may be placed at positions adjacent to or spaced apart from the shown sensor (162).

[0162] FIG. 10 is an exemplary diagram showing a door control module (150) of a refrigerator (1) according to one example of the present disclosure.

[0163] Referring to FIG. 10, the refrigerator (1) may include a door control module (150). The door control module (150) may control the opening or closing of at least one door. The door control module (150) may include a motor drive unit (151) and a motor (152). For example, the door control module (150) may precisely control the movement of at least one door (300) according to whether or not the at least one door (300) is open or the degree of opening.

[0164] The motor drive unit (151) can control the motor. For example, the motor drive unit (151) can activate or deactivate the motor (152). For example, the motor drive unit (151) can control the operating state of the motor (152) by supplying or cutting off power to the motor (152).

[0165] The motor (152) can open at least one door by rotating. For example, the motor (152) can open at least one door by being activated based on the control of the motor drive unit. For example, the motor (152) can close at least one door by being deactivated based on the control of the motor drive unit.

[0166] As shown in FIG. 10, the door control module (150) may include a plurality of door control modules corresponding to each of at least one door (300). For example, the door control module (150) may include a first door control module (150a) to a fourth door control module (150d). Each of the first door control module (150a) to the fourth door control module (150d) may include a motor drive unit (151) and a motor (152) for controlling the opening and closing of at least one door (300).

[0167] The first door control module (150a) is formed on an upper frame adjacent to the left upper door (300a) so as to control the opening or closing of the left upper door (300a). For example, the first motor drive unit (151a) receives a door control signal from the processor (191) and can control the opening or closing of the left upper door (300a) by activating or deactivating the first motor (152a) based on the door control signal.

[0168] The second door control module (150b) is formed on an upper frame adjacent to the right upper door (300b) so as to control the opening or closing of the right upper door (300b). For example, the second motor drive unit (151b) receives a door control signal from the processor (191) and controls the opening or closing of the right upper door (300b) by activating or deactivating the second motor (152b) based on the door control signal.

[0169] The third door control module (150c) is formed in the lower frame adjacent to the left lower door (300c) so as to control the opening or closing of the left lower door (300c). For example, the third motor drive unit (151c) receives a door control signal from the processor (191) and controls the opening or closing of the left lower door (300c) by activating or deactivating the third motor (152c) based on the door control signal.

[0170] The fourth door control module (150d) is formed in the lower frame adjacent to the right lower door (300d) so as to control the opening or closing of the right lower door (300d). For example, the fourth motor drive unit (151d) receives a door control signal from the processor (191) and controls the opening or closing of the right lower door (300d) by activating or deactivating the fourth motor (152d) based on the door control signal.

[0171] FIGS. 11a and 11b illustrate a method for opening the automatic door of a refrigerator (1) according to one example.

[0172] Referring to FIGS. 11a and 11b, the refrigerator (1) can identify a gesture of a user (1110) located within a first distance (1120) from the refrigerator (1). The refrigerator (1) can open a first door based on the identified gesture being identified as a first gesture.

[0173] In FIG. 11a, the refrigerator (1) can identify a gesture of a user (1110) located within a first distance (1120) from the refrigerator (1). The refrigerator (1) can acquire point cloud data using at least one sensor (162). The point cloud data may include information regarding the movement of a plurality of points corresponding to an object located within the first distance (1120) from the refrigerator (1). The refrigerator (1) can identify a gesture of a user (1110) located within the first distance (1120) from the refrigerator (1) based on the acquired point cloud data. For example, as described with reference to FIG. 6 and 7, the refrigerator (1) can identify a gesture of a user (1110) based on the point cloud data. For example, the refrigerator (1) can cluster the point cloud of the user (1110) from the point cloud data. The refrigerator (1) can detect the skeleton of the user (1110) from the point cloud of the user (1110). The refrigerator (1) can track the movement of the skeleton. The refrigerator (1) can identify the user's gesture based on the movement of the skeleton. For example, the user (1110) may make a gesture of raising their hand (left hand and / or right hand) from bottom to top above a threshold. The refrigerator (1) can identify the gesture of raising the hand from bottom to top above a threshold.

[0174] In FIG. 11b, the refrigerator (1) can open the first door based on the identified gesture being identified as the first gesture. The first gesture may be a pre-set gesture to open the first door. For example, the first gesture may be a gesture of raising a hand from bottom to top. The refrigerator (1) can infer the gesture intended to be provided by the user (e.g., the first gesture) based on point cloud data using an artificial intelligence model trained to identify the user's gesture. The refrigerator (1) can open the first door based on identifying the first gesture. For example, the refrigerator (1) can pre-set the first door. For example, the refrigerator (1) can set the upper door (300a; 300b) as the first door. For example, the refrigerator (1) can set the first door according to user input. For example, the refrigerator (1) can set the upper door (300a or 300b) that is close to the hand raised from bottom to top according to user input as the first door. The refrigerator (1) can provide a user interface that can set the first door.

[0175] FIGS. 12a and 12b illustrate a method for opening the automatic door of a refrigerator (1) according to one example.

[0176] Referring to FIGS. 12a and 12b, the refrigerator (1) can identify a gesture of a user (1210) located within a first distance (1220) from the refrigerator (1). The refrigerator (1) can open a first door based on the identified gesture being identified as a first gesture.

[0177] In FIG. 12a, the refrigerator (1) can identify a gesture of a user (1210) located within a first distance (1220) from the refrigerator (1). For example, the user (1210) may make a gesture of moving their foot. The refrigerator (1) can identify a gesture of moving the foot. Since the gesture identification method in FIG. 12a is substantially the same or redundant as the gesture identification method in FIG. 11a, the description is omitted.

[0178] In FIG. 12b, the refrigerator (1) can open the first door based on the identified gesture being identified as the first gesture. For example, the first gesture may be a gesture of moving the foot. For example, the first gesture may be a gesture of moving the foot from left to right. For example, the first gesture may be a gesture of moving the foot from right to left. The refrigerator (1) can pre-set the first door. For example, the refrigerator (1) can set the right upper door (300 (12)) as the first door. The refrigerator (1) can set the first door according to user input. For example, the refrigerator (1) can set the door on the side where the foot movement stopped as the first door according to user input. For example, if the first gesture is a gesture of moving the foot from left to right, the door on the right where the foot stopped (e.g., the right upper door (300 (12))) can be set as the first door. The refrigerator (1) can provide a user interface that can set the first door.

[0179] For example, the user (1210) may be holding luggage in their hands. The user (1210) can conveniently open the door using their feet even while holding luggage in their hands.

[0180] FIGS. 13a and 13b illustrate a method for opening the automatic door of a refrigerator (1) according to one example.

[0181] Referring to FIGS. 13a and 13b, the refrigerator (1) can identify a gesture of a user (1310) located within a first distance (1320) from the refrigerator (1). The refrigerator (1) can open a first door based on the identified gesture being identified as a first gesture.

[0182] In FIG. 13a, the refrigerator (1) can identify a gesture of a user (1310) located within a first distance (1320) from the refrigerator (1). For example, the user (1310) may make a gesture of reaching out a hand toward the door. The refrigerator (1) can identify a gesture of reaching out a hand toward the door. Since the gesture identification method in FIG. 13a is substantially the same or redundant as the gesture identification method in FIG. 11a, the description is omitted.

[0183] In FIG. 13b, the refrigerator (1) can open the first door based on the identified gesture being identified as the first gesture. For example, the first gesture may be a gesture of reaching out a hand toward the door. The refrigerator (1) can pre-set the first door. For example, the refrigerator (1) can set the right upper door (300b) as the first door. The right upper door (300b) may include an inner door (110) and an outer door (120). For example, the refrigerator (1) can set the inner door (110) and / or the outer door (120) as the first door. For example, the refrigerator (1) can open the inner door (110) and / or the outer door (120) in response to the first gesture. The refrigerator (1) can set the first door according to user input. For example, the refrigerator (1) can set the door toward which the hand is directed as the first door. For example, if the first gesture is a gesture of reaching out a hand toward the right upper door (300b), the right upper door (300b) can be set as the first door. The refrigerator (1) can provide a user interface that can set the first door.

[0184] The user (1310) can conveniently open the door using an intuitive gesture (e.g., a gesture of reaching out a hand toward the door).

[0185] FIG. 14 is a flowchart illustrating the operation of a refrigerator according to one example of the present disclosure.

[0186] In the following examples of operations, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of the operations may be changed, or at least two operations may be performed in parallel. At least some of the operations may be performed on a server.

[0187] In operation 1410, the refrigerator (1) can identify a gesture of a user located within a first distance from the refrigerator. In operation 1420, the refrigerator (1) can determine whether the identified gesture is the first gesture. In operation 1430, if the identified gesture is the first gesture, the refrigerator (1) can open the first door. If the identified gesture is not the first gesture, the refrigerator (1) can return to operation 1410. Since operations 1410 to 1430 are substantially identical to or duplicate operations 510 to 530 of FIG. 5, the description is omitted.

[0188] In operation 1440, the refrigerator (1) can determine whether a user is located within a first distance from the refrigerator. The refrigerator (1) can identify a user located within a first distance from the refrigerator. For example, the refrigerator (1) can acquire point cloud data using at least one sensor (e.g., sensor (162) of FIG. 4). Based on the acquired point cloud data, the refrigerator (1) can identify a user located within a first distance from the refrigerator. If the refrigerator (1) does not identify a user located within a first distance from the refrigerator, it can determine that the user is not located within a first distance from the refrigerator.

[0189] When a user located within a first distance from the refrigerator is identified, the refrigerator (1) may, in operation 1440, identify the user's gesture and determine whether the identified gesture is a second gesture. The second gesture may be a pre-set gesture to close at least one door included in the refrigerator. In operation 1440, the method by which the refrigerator (1) identifies the gesture and determines the gesture intended to be provided by the user (e.g., the second gesture) is substantially the same or redundant as the gesture identification method in operations 520 and 530 of FIG. 5, so a description is omitted. If the identified gesture is not the second gesture, the refrigerator may return to operation 1410.

[0190] In operation 1450, the refrigerator (1) may close at least one door included in the refrigerator (1) if the user is not located within a first distance from the refrigerator. The refrigerator (1) may close at least one door included in the refrigerator (1) if it identifies that the user is not located within a first distance from the refrigerator for more than a specified time (e.g., 3 seconds). In operation 1450, the refrigerator (1) may close at least one door included in the refrigerator (1) if the identified gesture is a second gesture. The door to be closed may be the first door. The door to be closed may be any door that is open in the refrigerator (1).

[0191] In one embodiment, the refrigerator (1) comprises at least one storage room (20); at least one door (300) for opening and closing the at least one storage room; a door control module (150) for controlling the opening or closing of the at least one door; at least one sensor (162); and at least one processor (191). The at least one processor may be configured to acquire point cloud data using the at least one sensor, identify a gesture of a user located within a first distance from the refrigerator based on the point cloud data, and control the door control module to open a first door selected from the at least one door based on the identification of the identified gesture as a first gesture.

[0192] In one embodiment, the at least one processor may be configured to cluster the user's point cloud from the point cloud data, detect the user's skeleton from the user's point cloud, track the movement of the skeleton, detect features of the skeleton's movement, and identify the user's gesture based on the features of the skeleton's movement.

[0193] In one embodiment, the at least one processor may be configured to learn the characteristics of the user's skeleton and, based on the characteristics of the skeleton, identify the user's gestures from the point cloud data.

[0194] In one embodiment, the at least one processor may be configured to identify the user's gesture by inferring the gesture intended to be provided by the user based on the point cloud data using an artificial intelligence model trained to identify the user's gesture.

[0195] In one embodiment, the at least one processor may be configured to identify a gesture of the user after identifying that the user is located within the first distance for more than a specified time based on the point cloud data.

[0196] In one embodiment, the at least one processor may be configured to provide a user interface that can select the first door to be opened in response to the first gesture among the at least one doors.

[0197] In one embodiment, the at least one processor may be configured to detect a restricted user based on the point cloud data, and to disable the door opening function when it detects the first gesture of the restricted user.

[0198] In one embodiment, the at least one processor may be configured to control the door control module to close the at least one door based on the identification that the user is not located within a first distance from the refrigerator.

[0199] In one embodiment, the at least one processor may be configured to control the door control module to close the at least one door based on the user's gesture being identified as a second gesture.

[0200] In one embodiment, the at least one sensor may include at least one of a radar sensor, an optical sensor, or an ultrasonic sensor.

[0201] In one embodiment, a control method for a refrigerator (1) may include: acquiring point cloud data using at least one sensor; identifying a gesture of a user located within a first distance from the refrigerator based on the point cloud data; and opening a first door based on the identification of the identified gesture as a first gesture.

[0202] In one embodiment, the operation of identifying the user's gesture can cluster the user's point cloud from the point cloud data, detect the user's skeleton from the user's point cloud, track the movement of the skeleton, detect the characteristics of the skeleton's movement, and identify the user's gesture based on the characteristics of the skeleton's movement.

[0203] In one embodiment, the operation of identifying the user's gesture learns the characteristics of the user's skeleton and, based on the characteristics of the skeleton, can identify the user's gesture from the point cloud data.

[0204] In one embodiment, the operation of identifying the user's gesture may include the operation of inferring the gesture intended by the user based on the point cloud data using an artificial intelligence model trained to identify the user's gesture.

[0205] In one embodiment, the operation of identifying the user's gesture can identify the user's gesture after identifying that the user is located within the first distance for more than a specified time based on the point cloud data.

[0206] In one embodiment, the control method of the refrigerator may include an operation of providing a user interface that can select the first door to be opened in response to the first gesture.

[0207] In one embodiment, the control method of the refrigerator may include: an operation of detecting a restricted user based on the point cloud data; and an operation of disabling the door opening function when the first gesture of the restricted user is detected.

[0208] In one embodiment, the control method of the refrigerator may include closing at least one door based on identifying that the user is not located within a first distance from the refrigerator.

[0209] In one embodiment, the control method of the refrigerator may include the action of closing at least one door based on the user's gesture being identified as a second gesture.

[0210] In one embodiment, in the control method of the refrigerator, the at least one sensor may include at least one of a radar sensor, a light sensor, or an ultrasonic sensor.

[0211] In one embodiment, the refrigerator comprises at least one storage compartment; at least one door configured to open and close the at least one storage compartment; at least one sensor; and at least one processor (191), wherein the at least one processor may be configured to acquire point cloud data using the at least one sensor, identify a gesture of a user located within a first distance from the refrigerator based on the point cloud data, and perform control to open one of the at least one doors based on the identified gesture.

[0212] In one embodiment, the at least one processor may be configured to cluster the user's point cloud from the point cloud data, detect the user's skeleton from the user's point cloud, track the movement of the skeleton, detect features of the skeleton's movement, and identify the user's gesture based on the features of the skeleton's movement.

[0213] In one embodiment, the at least one processor may be configured to learn the characteristics of the user's skeleton and, based on the characteristics of the skeleton, identify the user's gestures from the point cloud data.

[0214] In one embodiment, the at least one processor may be configured to identify the user's gesture by inferring the gesture intended to be provided by the user based on the point cloud data using an artificial intelligence model trained to identify the user's gesture.

[0215] In one embodiment, the at least one processor may be configured to identify a gesture of the user after identifying that the user is located within the first distance for more than a specified time based on the point cloud data.

[0216] In one embodiment, the at least one door includes a plurality of doors, and the at least one processor may be configured to provide a user interface that can select the door to be opened among the plurality of doors in response to the gesture.

[0217] In one embodiment, the at least one processor may be configured to detect a restricted user based on point cloud data acquired using the at least one sensor, and to disable the door opening function when it detects the gesture of the restricted user.

[0218] In one embodiment, the at least one processor may be configured to perform control to close the door based on identifying that the user is not located within the first distance from the refrigerator while the door is open.

[0219] In one embodiment, the gesture is a first gesture, and the at least one processor may be configured to acquire additional cloud point data using the at least one sensor while the door is open, identify a second gesture of the user located within the first distance from the refrigerator based on the additional cloud point data, and perform control to close the door based on the identified second gesture of the user.

[0220] In one embodiment, the at least one sensor may include at least one of a radar sensor, an optical sensor, or an ultrasonic sensor.

[0221] In one embodiment, the at least one door includes a plurality of doors, the door is a first door among the plurality of doors, the gesture is a first gesture for opening the first door, and the at least one processor may be configured to identify a second gesture of the user located within a first distance from the refrigerator based on cloud point data acquired using the at least one sensor, the second gesture is different from the first gesture, the second gesture is a gesture for opening a second door among the plurality of doors, and the processor may be configured to perform control to open the second door based on the identified second gesture.

[0222] In one embodiment, a method for controlling a refrigerator may include: acquiring point cloud data using at least one sensor; identifying a gesture of a user located within a first distance from the refrigerator based on the point cloud data; and opening the door of the refrigerator based on the identified gesture.

[0223] In one embodiment, the operation of identifying the user’s gesture may include the operation of clustering the user’s point cloud from the point cloud data, the operation of detecting the user’s skeleton from the user’s point cloud, the operation of tracking the movement of the skeleton, the operation of detecting the characteristics of the skeleton’s movement, and the operation of identifying the user’s gesture based on the characteristics of the skeleton’s movement.

[0224] In one embodiment, the operation of identifying the user’s gesture may include the operation of learning the characteristics of the user’s skeleton, and the operation of identifying the user’s gesture from the point cloud data based on the characteristics of the skeleton.

[0225] In one embodiment, the operation of identifying the user's gesture may include the operation of inferring the gesture intended by the user based on the point cloud data using an artificial intelligence model trained to identify the user's gesture.

[0226] In one embodiment, the operation of identifying the user's gesture may include identifying the user's gesture after identifying that the user is located within the first distance for a specified amount of time based on the point cloud data.

[0227] In one embodiment, the refrigerator includes a plurality of doors, and the method may further include an operation of providing a user interface that can select the door to be opened among the plurality of doors in response to the gesture.

[0228] In one embodiment, the method may include an operation to detect a restricted user based on point cloud data acquired using at least one sensor, and an operation to disable a door opening function when the gesture of the restricted user is detected.

[0229] In one embodiment, the method may further include the operation of closing the door based on the identification that the user is not located within the first distance from the refrigerator while the door is open.

[0230] In one embodiment, the gesture is a first gesture, and the method may include acquiring additional cloud point data using at least one sensor while the door is open, identifying a second gesture of the user located within a first distance from the refrigerator based on the additional cloud point data, and closing the door based on the identified second gesture of the user.

[0231] In one embodiment, the at least one sensor may include at least one of a radar sensor, an optical sensor, or an ultrasonic sensor.

[0232] In one embodiment, the refrigerator includes a plurality of doors, the door is a first door among the plurality of doors, and the gesture is a first gesture for opening the first door, and the method may include an action of identifying a second gesture of a user located within a first distance from the refrigerator based on cloud point data acquired using at least one sensor, wherein the second gesture is different from the first gesture and the second gesture is a gesture for opening a second door among the plurality of doors, and an action of opening the second door based on the identified second gesture.

[0233] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0234] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In the refrigerator (1), At least one storage room (20); At least one door (300) configured to open and close at least one storage room; At least one sensor (162); and It includes at least one processor (191), The above-mentioned at least one processor is, Point cloud data is acquired using at least one sensor as described above, and Identifying a user's gesture located within a first distance from the refrigerator based on the above point cloud data, and Configured to perform control to open one of the at least one door based on the identified gesture, refrigerator.

2. In Paragraph 1, The above-mentioned at least one processor is, Clustering the user's point cloud from the above point cloud data, and Detecting the user's skeleton from the user's point cloud, and Tracking the movement of the above skeleton, Detecting the characteristics of the movement of the above skeleton, and Configured to identify the user's gestures based on the movement characteristics of the skeleton, refrigerator.

3. In Paragraph 2, The above-mentioned at least one processor is, Learning the characteristics of the above user's skeleton, and Based on the characteristics of the above skeleton, configured to identify the user's gesture from the point cloud data, refrigerator.

4. In any one of paragraphs 1 through 3, The above-mentioned at least one processor is, A configuration for identifying the user's gesture by inferring the gesture intended to be provided by the user based on the point cloud data using an artificial intelligence model trained to identify the user's gesture, refrigerator.

5. In any one of paragraphs 1 through 4, The above-mentioned at least one processor is, Configured to identify the user's gesture after identifying that the user is located within the first distance for a specified time or longer based on the above point cloud data, refrigerator.

6. In any one of paragraphs 1 through 5, The above at least one door includes a plurality of doors, and The above-mentioned at least one processor is, A user interface configured to select the door to be opened among the plurality of doors in response to the above gesture, refrigerator.

7. In any one of paragraphs 1 through 6, The above-mentioned at least one processor is, Based on point cloud data obtained using the above at least one sensor, a restricted user is detected, and Configured to disable the door opening function when the above-mentioned restricted user's gesture is detected, refrigerator.

8. In any one of paragraphs 1 through 7, The above at least one processor, when the door is open, Configured to perform control to close the door based on the identification that the user is not located within the first distance from the refrigerator. refrigerator.

9. In any one of paragraphs 1 through 8, The above gesture is the first gesture, and The above at least one processor, when the door is open, Additional cloud point data is obtained using at least one of the above sensors, and Based on the additional cloud point data above, identify the second gesture of the user located within the first distance from the refrigerator, and Configured to perform control to close the door based on the second gesture of the identified user, refrigerator.

10. In any one of paragraphs 1 through 9, The above at least one sensor comprises at least one of a radar sensor, an optical sensor, or an ultrasonic sensor. refrigerator.

11. In any one of paragraphs 1 through 10, The above at least one door includes a plurality of doors, and The above door is the first door among the plurality of doors, and The above gesture is a first gesture for opening the first door, and The above-mentioned at least one processor is, Based on cloud point data acquired using at least one sensor, a second gesture of the user located within the first distance from the refrigerator is identified, the second gesture is different from the first gesture, and the second gesture is a gesture for opening the second door among the plurality of doors. Configured to perform control to open the second door based on the second gesture identified above, refrigerator.

12. In a method for controlling a refrigerator (1), The operation of acquiring point cloud data using at least one sensor; An action of identifying a user's gesture located within a first distance from the refrigerator based on the above point cloud data; and The action of opening the refrigerator door based on the above-identified gesture A method for controlling a refrigerator, including 13. In Paragraph 12, The action of identifying the above user's gesture is, The operation of clustering the user's point cloud from the above point cloud data, The operation of detecting the user's skeleton from the user's point cloud, Actions for tracking the movement of the above skeleton, An operation for detecting the characteristics of the movement of the above-mentioned skeleton, and A method including an action for identifying the user's gesture based on the movement characteristics of the skeleton. Refrigerator control method.

14. In Paragraph 13, The action of identifying the above user's gesture is, The operation of learning the characteristics of the skeleton of the above user, and Based on the characteristics of the above skeleton, the operation of identifying the user's gesture from the point cloud data, Refrigerator control method.

15. In any one of paragraphs 12 through 14, The action of identifying the above user's gesture is, A motion including inferring the gesture intended to be provided by the user based on the point cloud data using an artificial intelligence model trained to identify the gesture of the user. Refrigerator control method.

Images