Mobile robot for providing output information on basis of user detection information, and operation method thereof

Abstract

The present invention relates to a mobile robot for providing output information on the basis of user detection information, and an operation method thereof. The mobile robot according to one aspect of the present invention comprises: a camera unit for photographing the front of the mobile robot; a user interface unit including a speaker and / or a display; and a processor unit that, when a user is included in an image transmitted from the camera unit, generates user detection information about the user on the basis of the image and a pre-generated map and outputs output information corresponding to the generated user detection information.

Description

A mobile robot that provides output information based on user detection information and a method of operation thereof

[0001] The present invention relates to a mobile robot that provides output information based on user detection information and a method of operation thereof, and more specifically, to a mobile robot that provides output information based on user detection information optimized for the user and a method of operation thereof.

[0002] Mobile robots move freely within indoor spaces and provide services optimized for users through user interfaces (UI), such as speakers and displays.

[0003] For example, when a mobile robot receives a user command, it provides output information to the user by outputting sound, images, or text based on the user's command. Additionally, for example, while moving within an indoor space, the mobile robot can detect an abnormal situation (e.g., fire) and provide a notification to the user through a speaker to alert them of the occurrence of the abnormal situation.

[0004] In this regard, in the case of the prior art, output information is provided to the user in a manner similar to a user terminal (e.g., smartphone, tablet) used at a close distance to the user, without considering the distance between the user and the mobile robot.

[0005] For example, when the mobile robot receives a user command for news streaming, it streams the news through a speaker while simultaneously outputting text through a display to provide information to the user. However, the distance between the user and the mobile robot changes as it frequently moves further away or closer.

[0006] As a result, in the case of conventional technology, information is provided without considering the distance between the mobile robot and the user, so when the distance between the user and the mobile robot is long, text readability decreases, leading to a problem where the user cannot accurately receive the output information provided by the mobile robot.

[0007] In addition, for example, when the mobile robot detected an abnormal situation, it provided a notification to the user through a speaker.

[0008] As a result, in the case of conventional technology, there was a problem in that mobile robots only provide notifications to the user about the occurrence of abnormal situations, but it was difficult to provide output information to the user that could respond to the abnormal situation.

[0009] The present invention was devised to solve the problems described above, and the objective of the present invention is to provide a mobile robot and a method of operation thereof that provides output information based on user detection information capable of estimating the distance between a user and a mobile robot and providing information optimized to the user based on the estimated distance.

[0010] Another objective of the present invention is to provide a mobile robot and a method of operation thereof that estimates a user's location area and provides output information based on user detection information capable of providing information optimized for the user based on the estimated user's location area.

[0011] Another objective of the present invention is to provide a mobile robot and a method of operation thereof that recognizes a user's face, estimates the angle of the recognized user's face, and provides output information based on user detection information capable of providing information optimized for the user based on the estimated angle of the user's face.

[0012] The objectives of the present invention are not limited to the problems mentioned above, and other objectives and advantages of the present invention not mentioned may be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be understood that the objectives and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0013] A mobile robot that provides output information based on user detection information according to the present invention for solving the problem described above is characterized by the technical feature of providing information optimized for the user based on user detection information detected about the user.

[0014] Specifically, the mobile robot detects user detection information based on video, including the distance between the user and the mobile robot, the user's location area, and the angle of the user's face. By setting an information representation method based on the detected user detection information, even if the mobile robot performs the same function, output information is provided based on the user detection information, thereby enabling the provision of information optimized for the user.

[0015] For the above purpose, a mobile robot according to one embodiment of the present invention comprises: a camera unit for photographing the front of the mobile robot; a user interface unit including at least one of a speaker and a display; and a processor unit that, when a user is included in an image transmitted from the camera unit, generates user detection information for the user based on the image and a previously generated map, and outputs output information corresponding to the generated user detection information.

[0016] Preferably, the processor unit detects a human shape and a face in the image, estimates the distance between the mobile robot and the user when the detected face matches the previously stored user information, and generates user detection information by estimating the area where the user is located based on the map and the estimated distance between the mobile robot and the user.

[0017] Preferably, the processor unit generates user detection information regarding whether the estimated user location area and the mobile robot location area are the same based on the estimated user location area, and outputs output information based on whether the estimated user location area and the mobile robot location area are the same.

[0018] Preferably, the processor unit generates user detection information by estimating the angle of the user's face based on at least one angle among the user's nose and mouth when the distance between the estimated mobile robot and the user is less than a distance threshold, and outputs output information based on the estimated angle of the user's face.

[0019] Meanwhile, a method of operation of a mobile robot according to one embodiment of the present invention comprises: a step of photographing the front of the mobile robot; a step of generating user detection information for the user based on the image and a previously generated map when the image transmitted from the camera unit includes a user; and a step of outputting output information corresponding to the generated user detection information.

[0020] Preferably, the step of generating the user detection information comprises: detecting a human shape and a face in the image; estimating the distance between the mobile robot and the user when the detected face matches the previously stored user information; and estimating the area where the user is located based on the estimated distance between the mobile robot and the user.

[0021] Preferably, the step of generating the user detection information includes the step of generating user detection information regarding whether the estimated user location area and the mobile robot location area are the same based on the estimated user location area, and the step of outputting the output information includes the step of outputting the output information based on whether the estimated user location area and the mobile robot location area are the same.

[0022] Preferably, the step of generating the user detection information includes the step of estimating the angle of the user's face based on at least one angle of the user's nose and mouth when the distance between the estimated mobile robot and the user is less than a distance threshold, and the step of outputting the output information includes the step of outputting the output information based on the estimated angle of the user's face.

[0023] According to the present invention, user detection information regarding the distance between a mobile robot and a user is generated, and output information is provided based on the generated user detection information, thereby having the effect of effectively providing information to the user.

[0024] Furthermore, according to the present invention, the area where a user is located is estimated to generate user detection information, and output information is provided based on the generated user detection information, thereby having the effect of effectively providing information to the user.

[0025] Furthermore, according to the present invention, the user's face is detected, the angle of the detected user's face is estimated to generate user detection information, and output information is provided based on the generated user detection information, thereby having the effect of effectively providing information to the user.

[0026] FIG. 1 is a configuration diagram of a mobile robot that provides output information based on user detection information according to one embodiment of the present invention.

[0027] FIG. 2 is a flowchart of a method of operation of a mobile robot that provides output information based on user detection information according to an embodiment of the present invention.

[0028] Figure 3 is a detailed flowchart for step S220 of Figure 2.

[0029] FIG. 4 is an example diagram of a case where a mobile robot and a user are located in the same area according to an embodiment of the present invention.

[0030] FIG. 5 is an example diagram of a case where a mobile robot and a user are located in different areas according to an embodiment of the present invention.

[0031] FIG. 6 is a conceptual diagram illustrating output information according to the distance between a mobile robot and a user according to an embodiment of the present invention.

[0032] FIG. 7 is an exemplary diagram illustrating text output according to the angle of a user's face according to an embodiment of the present invention.

[0033] FIG. 8 is an exemplary diagram illustrating image output according to the angle of a user's face according to one embodiment of the present invention.

[0034] FIG. 9 is a conceptual diagram illustrating output information according to the angle of a user's face according to an embodiment of the present invention.

[0035] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. However, in describing the present invention, descriptions of already known functions or configurations will be omitted in order to clarify the gist of the present invention.

[0036] Hereinafter, a mobile robot that provides information based on user detection information according to the present invention and a method of operation thereof will be described in detail with reference to FIGS. 1 to 9.

[0037] FIG. 1 is a configuration diagram of a mobile robot that provides information based on user detection information according to an embodiment of the present invention, FIG. 2 is a flowchart of an operation method of a mobile robot that provides information based on user detection information according to an embodiment of the present invention, and FIG. 3 is a detailed flowchart of step S220 of FIG. 2.

[0038] Referring to FIG. 1, a mobile robot (100) that provides information based on user detection information according to one embodiment of the present invention includes a camera unit (110), a user interface unit (120), a processor unit (130), etc.

[0039] The camera unit (110) photographs the front of the mobile robot (100) (see step S210 of FIG. 2).

[0040] The camera unit (110) includes a depth sensor and an image sensor.

[0041] And, the camera unit (110) acquires depth information based on images captured through a plurality of sensors.

[0042] For example, the depth sensor and the image sensor may be positioned at different locations on the same vertical line. Accordingly, when the depth sensor and the image sensor simultaneously capture an object at the same location, the camera unit (110) aligns the pixel coordinate values ​​of the object images captured by each sensor to obtain depth information. The processor unit (130) can generate user detection information using the depth information obtained from the camera unit (110).

[0043] The mobile robot (100) provides output information to the user (200) through the user interface unit (120).

[0044] At this time, the output information refers to information provided to the user (200) while interacting with the mobile robot (100), the user (200), and other devices.

[0045] The user interface section (120) includes at least one of a speaker and a display.

[0046] For example, the mobile robot (100) can provide information to the user (200) by outputting sound through a speaker and outputting at least one of an image and text through a display.

[0047] When the user (200) is included in the image transmitted from the camera unit (110), the processor unit (130) generates user detection information for the user (200) based on the image and a previously generated map (see step S220 of FIG. 2).

[0048] User detection information includes whether the location area of ​​the mobile robot (100) matches the estimated user location area, the distance between the mobile robot (100) and the user (200), and the angle of the user's face (200).

[0049] First, the processor unit (130) detects the user (200) based on the image captured from the camera unit (110).

[0050] For example, when the processor unit (130) detects a human form (e.g., skeleton form) from a captured image, it can detect a user (200) by determining whether the detected human face matches the previously stored user information (e.g., face ID).

[0051] And, the processor unit (130) estimates the distance between the mobile robot (100) and the detected user (200) based on depth information about the detected user (200) obtained from the camera unit (110).

[0052] Then, the processor unit (130) estimates the area where the user (200) is located based on the previously generated map, the location of the mobile robot (100), and the estimated distance between the mobile robot (100) and the user (200) (see step S221 of FIG. 2).

[0053] For example, the processor unit (130) estimates the location of the user (200) by adding the distance between the mobile robot (100) and the user (200) estimated at the location of the mobile robot (100). Then, the processor unit (130) estimates the area corresponding to the estimated location of the user (200) on the previously generated map as the user location area.

[0054] At this time, the map can be generated based on an image captured from the camera unit (110).

[0055] For example, a mobile robot (100) extracts multiple feature points from a video captured while moving. Then, the mobile robot (100) can recognize the initial position of the mobile robot by obtaining position information of the mobile robot (100) through tracking the extracted multiple feature points. Then, the mobile robot (100) generates a map of the space based on the recognized initial position.

[0056] In this case, a feature point refers to a point that can serve as a comparison point to compare points that have changed according to the movement of the mobile robot.

[0057] And, the mobile robot (100) can divide the generated map into multiple regions based on the passages within the generated map.

[0058] For example, the mobile robot (100) divides the map generated based on the end of the passage into multiple regions.

[0059] Additionally, when a map generated by the mobile robot (100) is transmitted to a user terminal linked to the mobile robot (100), the user (200) can edit the generated map through the user terminal.

[0060] For example, the user (200) can adjust spaces and areas on a map generated through a user terminal, or edit the names of the separated areas. Then, the edited map is transmitted to and stored by the mobile robot (100).

[0061] The processor unit (130) determines whether the location area of ​​the mobile robot (100) matches the estimated user location area (see step S222 of FIG. 3).

[0062] In this regard, FIG. 4 is an example diagram of a case where a mobile robot and a user are located in the same area according to an embodiment of the present invention.

[0063] For example, if we assume that the area where the mobile robot (100) is located and the area where the estimated user (200) is located are a 'living room', the processor unit (130) determines that the area where the mobile robot (100) and the user (200) are located coincides.

[0064] Additionally, if it is determined that the mobile robot (100) location area and the user location area are the same, the processor unit (130) determines whether the distance between the estimated mobile robot (100) and the user (200) is less than a distance threshold (see step S223 of FIG. 3).

[0065] At this time, the distance threshold can be implemented as a distance at which the user (200) can have close-range interaction (e.g., touch, voice recognition, etc.) with the mobile robot (100) without moving the positions of the mobile robot (100) and the user (200).

[0066] If the estimated distance between the mobile robot (100) and the user (200) is less than a distance threshold, the processor unit (130) estimates the face angle of the user (200) (see step S224 of FIG. 3).

[0067] For example, the processor unit (130) estimates the angle of the user's (200) face based on the angle of the user's (200) nose and the angle of the user's (200) mouth.

[0068] Therefore, the gaze of the user (200) can be estimated through the estimated angle of the user's (200) face.

[0069] In this regard, FIG. 5 is an example diagram of a case where a mobile robot and a user are located in different areas according to an embodiment of the present invention.

[0070] For example, assuming that the area where the mobile robot (100) is located is the 'living room' and the area where the estimated user (200) is located is the 'bedroom', the processor unit (130) determines that the areas where the mobile robot (100) and the user (200) are located do not match.

[0071] And, the processor unit (130) outputs output information based on whether the location area of ​​the mobile robot (100) matches the estimated user (200) location area, the distance between the estimated mobile robot (100) and the user (200), and the face angle of the user (200) (see step S230 of FIG. 2).

[0072] When the mobile robot (100) and the user (200) are in different areas, the processor unit (130) controls the user interface unit (120) to output sound.

[0073] For example, assuming that the mobile robot (100) and the user (200) are in different areas and the mobile robot (100) detects an abnormal situation, the processor unit (130) can control the user interface unit (120) to output a sound to provide the user (200) with a notification of the occurrence of an abnormal situation.

[0074] Therefore, the mobile robot (100) can provide information to the user (200) through sound even when the user (200) cannot see the display.

[0075] Meanwhile, when the mobile robot (100) and the user (200) exist in the same space, the processor unit (130) outputs output information based on at least one user detection information among the estimated distance between the mobile robot (100) and the user (200) and the face angle of the user (200).

[0076] If the distance between the estimated mobile robot (100) and the user (200) is greater than or equal to a distance threshold, the processor unit (130) outputs output information based on the distance between the estimated mobile robot (100) and the user (200).

[0077] In this regard, FIG. 6 is a conceptual diagram illustrating output information according to the distance between a mobile robot and a user according to one embodiment of the present invention.

[0078] For example, when a user (200) enters an area where a mobile robot is located, the processor unit (130) can output an icon of an abnormal situation (see FIG. 6a) as output information.

[0079] And, for example, when the distance between the mobile robot (100) and the user (200) gradually decreases, an image of the abnormal situation (see FIG. 6b) is output as output information, and as the distance between the mobile robot (100) and the user (200) decreases, the image can be enlarged and output.

[0080] And, when the distance between the mobile robot (100) and the user (200) is less than the distance threshold, the processor unit (130) can output an emergency call list (see FIG. 6c) as output information.

[0081] At this time, for example, the distance threshold can be implemented as a distance at which the user's (200) voice can be clearly recognized.

[0082] Therefore, the user (200) can quickly respond to an abnormal situation by selecting one of the output emergency call lists and making an emergency call.

[0083] Additionally, for example, when the distance between the mobile robot (100) and the user (200) is greater than or equal to a distance threshold, the processor unit (130) outputs the remaining battery status of the mobile robot (100) as an icon. On the other hand, when the distance between the mobile robot (100) and the user (200) is less than a distance threshold, the processor unit (130) outputs the remaining battery of the mobile robot (100) in the format of 'number %'.

[0084] Additionally, the processor unit (130) controls and outputs the brightness of the display and the volume of the speaker based on the distance between the mobile robot (100) and the user (200).

[0085] For example, the processor unit (130) can control and output the brightness of the display as the distance between the mobile robot (100) and the user (200) decreases. Additionally, the processor unit (130) can control and output the volume of the speaker as the distance between the mobile robot (100) and the user (200) increases.

[0086] Meanwhile, if the distance between the estimated mobile robot (100) and the user (200) is less than the distance threshold, the processor unit (130) outputs output information as an image or text based on the face angle of the estimated user (200).

[0087] In this regard, FIG. 7 is an exemplary diagram illustrating text output according to the angle of a user's face according to an embodiment of the present invention, and FIG. 8 is an exemplary diagram illustrating image output according to the angle of a user's face according to an embodiment of the present invention.

[0088] For example, assuming that the mobile robot (100) is outputting weather information, if the position and face angle of the user (200) correspond to the forward direction of the display, the weather information can be controlled to be output in the form of text (e.g., current weather, temperature, etc.).

[0089] On the other hand, if the position and face angle of the user (200) do not correspond to the forward direction of the display, the weather information can be controlled to be output in the form of an image (e.g., a clear weather icon).

[0090] Additionally, the output information can be moved up and down or left and right based on the estimated angle of the user's (200) face.

[0091] For example, output information displayed on the display can move up and down or left and right based on the angle of the user's (200) face.

[0092] In this regard, FIG. 9 is a conceptual diagram illustrating output information according to the angle of a user's face according to one embodiment of the present invention.

[0093] For example, if more text is displayed than can be displayed depending on the size of the display (see FIG. 9a), when the user (200) lowers their head to check the displayed text, the displayed information moves up and down (see FIG. 9b).

[0094] Therefore, when the distance between the mobile robot (100) and the user (200) is close, information optimized for the user (200) can be provided by taking into account the user's (200) gaze toward the mobile robot (100).

[0095] Accordingly, the mobile robot (100) can output output information based on user detection information between the mobile robot (100) and the user (200) and effectively deliver optimized information to the user (200) through the user interface unit (120).

Claims

1. As a mobile robot, A camera unit that films the front of a mobile robot; A user interface unit comprising at least one of a speaker and a display; and A processor unit comprising, when a user is included in an image transmitted from the camera unit, generating user detection information for the user based on the image and a previously generated map, and outputting output information corresponding to the generated user detection information. Mobile robot.

2. In Paragraph 1, The above processor unit is, Detecting human shapes and faces in the above image, and When the detected face matches the previously stored user information, the distance between the mobile robot and the user is estimated, and Generating user detection information by estimating the area where the user is located based on the map and the distance between the estimated mobile robot and the user. Mobile robot.

3. In Paragraph 2, The above processor unit is, Controlling and outputting the display brightness and speaker volume based on the estimated distance between the mobile robot and the user, Mobile robot.

4. In Paragraph 2, The above processor unit is, Based on the above estimated user location area, user detection information regarding whether the above estimated user location area and the above mobile robot location area are the same is generated, and Outputting output information based on whether the above-mentioned estimated user location area and the above-mentioned mobile robot location area are identical, Mobile robot.

5. In Paragraph 4, The above output information is, Based on whether the estimated user location area and the mobile robot location area are identical, implemented as at least one of sound, image, and text, Mobile robot.

6. In Paragraph 1, The above processor unit is, When the distance between the estimated mobile robot and the user is less than a distance threshold, user detection information is generated by estimating the user's face angle based on at least one angle among the user's nose and mouth, and Outputting output information based on the estimated user's face angle above, Mobile robot.

7. In Paragraph 6, The above output information is, Implemented as an image or text based on the above-mentioned estimated user face angle, Mobile robot.

8. In Paragraph 6, The above output information is, Moving up and down or left and right based on the above estimated user face angle, Mobile robot.

9. As a method of operation for a mobile robot, Step of photographing the front of the mobile robot; If a user is included in the image transmitted from the camera unit, a step of generating user detection information for the user based on the image and a previously generated map; and A step comprising outputting output information corresponding to the generated user detection information, Method of operation of a mobile robot.

10. In Paragraph 9, The step of generating the above user detection information is, A step of detecting human shapes and faces in the above image; A step of estimating the distance between the mobile robot and the user when the detected face matches the previously stored user information; and A step comprising estimating the area where the user is located based on the distance between the mobile robot and the user. Method of operation of a mobile robot.

11. In Paragraph 10, The step of outputting the above output information is, Controlling and outputting the display brightness and speaker volume based on the estimated distance between the mobile robot and the user, Method of operation of a mobile robot.

12. In Paragraph 10, The step of generating the user detection information includes the step of outputting user detection information regarding whether the estimated user location area and the mobile robot location area are the same based on the estimated user location area. The step of outputting the above output information includes the step of outputting the output information based on whether the estimated user location area and the mobile robot location area are identical. Method of operation of a mobile robot.

13. In Paragraph 12, The above output information is, Based on whether the estimated user location area and the mobile robot location area are identical, implemented as at least one of sound, image, and text, Method of operation of a mobile robot.

14. In Paragraph 9, The step of generating the user detection information includes, when the distance between the estimated mobile robot and the user is less than a distance threshold, the step of estimating the angle of the user's face based on at least one angle among the user's nose and mouth. The step of outputting the above output information includes the step of outputting the output information based on the estimated user's face angle. Method of operation of a mobile robot.

15. In Paragraph 14, The above output information is, Implemented as an image or text based on the above-mentioned estimated user face angle, Method of operation of a mobile robot.

16. In Paragraph 14, The above output information is, Moving up and down or left and right based on the above estimated user face angle, Method of operation of a mobile robot.

Images