Robot apparatus and control method thereof

Abstract

A robot apparatus is disclosed. The robot apparatus comprises a body, a plurality of robot arms, a sensor, a memory, and at least one processor. At least one instruction causes the robot apparatus to: identify a target on the basis of a sensing value of a sensor; perform a task related to the target by using at least one of the plurality of robot arms; when it is detected that the target is occluded by an object, control a first robot arm among the plurality of robot arms to grip the sensor and detach same from the body so as to change the position and sensing direction of the sensor; and identify a state of the target and the object on the basis of a sensing value of the sensor obtained at the changed position.

Description

Robot device and control method thereof

[0001] The present disclosure relates to a robot device and a method for controlling the same, and more specifically, to a robot device including a detachable robot head and a method for controlling the same.

[0002] With technological advancements, robotic technology is being utilized to replace human labor in various fields. In particular, robotic devices are being developed in sectors requiring delicate and precise tasks, such as factories, construction, medical sites, and aerospace.

[0003] A robot device may use at least one sensor for its operation. In the case of a typical robot device, the sensor is attached to the robot head side for use.

[0004] When conventional robot devices operate in confined spaces using only sensors attached to the robot head, there have been cases where objects could not be accurately identified due to occlusion. For example, a target may be obscured by surrounding objects.

[0005] A robot device according to at least one embodiment of the present disclosure comprises a main body, a plurality of robot arms mounted on the main body, a sensor connected to the main body in a detachable manner, a memory storing at least one instruction, and at least one processor configured to execute the at least one instruction. When the at least one instruction is executed individually or collectively by the at least one processor, the robot device identifies a target based on the sensing value of the sensor, performs a task related to the target using at least one of the plurality of robot arms, and when it is detected that the target is obscured by an object, controls a first robot arm among the plurality of robot arms to grasp the sensor and detach it from the main body to change the position and sensing direction of the sensor, and identifies the state of the target and the object based on the sensing value of the sensor obtained at the changed position.

[0006] When the above at least one instruction is executed individually or collectively by the above at least one processor, the robot device may control the second robot arm among the plurality of robot arms to grasp the target when the state of the target and the object is identified.

[0007] When the above at least one instruction is executed individually or collectively by the above at least one processor, the robot device may control the first robot arm to mount the sensor on the main body and grasp the target when the state of the target and the object is identified.

[0008] When the above at least one instruction is executed individually or collectively by the above at least one processor, the robot device may be configured to grasp and move the object using at least one of the plurality of robot arms before grasping the target if interference caused by the object occurs during the process of grasping the target.

[0009] The above at least one instruction, when executed individually or collectively by the at least one processor, may cause the robot device to move by using at least one of a plurality of robot arms to grasp the object before grasping the target, based on the object that interferes with the ability to grasp the target.

[0010] The above main body comprises a robot body, a driver disposed on the lower side of the robot body, and

[0011] It may include a robot head mounted on the upper side of the robot body. The plurality of robot arms may be coupled to both sides of the robot body. The sensor may be mounted on the robot head.

[0012] The robot head may further include a connection part configured to be connected to the sensor and a reel cable connected to the sensor. The reel cable may be extendable based on the state in which the sensor is separated from the main body.

[0013] The robot head may include a first interface configured to be connected to the sensor. Among the plurality of robot arms, the end effector of at least one first robot arm may include a second interface configured to be connected to the sensor. When the at least one instruction is executed individually or collectively by the at least one processor, the robot device may control the first robot arm to detach the sensor from the first interface and mount it to the second interface provided on the first robot arm when the occlusion is identified while the sensor is coupled to the robot head through the first interface.

[0014] When the above at least one instruction is executed individually or collectively by the above at least one processor, the robot device controls the first robot arm to sequentially move the sensor to a plurality of positions facing the target, and can identify the characteristics of the target and the characteristics of the target object by combining the sensing values ​​obtained from the plurality of positions.

[0015] A control method for a robot device according to at least one embodiment of the present disclosure comprises the steps of: controlling a first robot arm among a plurality of robot arms of the robot device to grasp a sensor of the robot device and detach it from the main body of the robot device when it is identified that a target is obscured by an object; moving the first robot arm to change the position and sensing direction of the sensor; identifying the state of the target and the object based on a sensing value obtained from the sensor at the changed position; and grasping the target using at least one of the plurality of robot arms.

[0016] The step of grasping the target using at least one of the plurality of robot arms may include the step of grasping the target by controlling a second robot arm among the plurality of robot arms while the sensor grasped by the first robot arm senses the target.

[0017] The step of grasping the target using at least one of the plurality of robot arms may include the step of controlling the first robot arm to remount the sensor to the main body when the state of the target and the object is identified based on the sensing value sensed by the sensor at the changed position, and the step of controlling the first robot arm to grasp the target.

[0018] The step of grasping the target using at least one of the plurality of robot arms may further include the step of grasping and moving the object first using at least one of the plurality of robot arms before grasping the target if interference with the object occurs during the process of grasping the target.

[0019] The step of grasping the target using at least one of the plurality of robot arms may include the step of grasping and moving the object using at least one of the plurality of robot arms before grasping the target, based on the object that interferes with the robot device's ability to grasp the target.

[0020] The step of controlling the first robot arm among the plurality of robot arms to grasp the sensor and detach it from the main body may include, when the occlusion is identified while the sensor is connected to a first interface provided on the robot head of the robot device, controlling the first robot arm to detach the sensor from the first interface mounted on the robot head of the robot device and connecting it to a second interface provided on the end effector of the first robot arm.

[0021] The step of moving the first robot arm to change the position and sensing direction of the sensor may include the step of controlling the first robot arm to sequentially move the sensor to a plurality of positions facing the target.

[0022] The step of identifying the state of the target and the object may include the step of identifying the characteristics of the target and the characteristics of the object by combining the total sensing values ​​sensed at the plurality of positions and directions.

[0023] A non-permanent computer-readable medium according to at least one embodiment of the present invention includes instructions stored in the medium, and when the instructions are executed by one or more processors, the processors execute a method of controlling a robot device. The method comprises the following steps: identifying that a target is obscured by an object, controlling a first robot arm among a plurality of robot arms of the robot device to grasp a sensor of the robot device and separating the sensor from the main body of the robot device; moving the first robot arm to change the position and detection direction of the sensor; identifying the state of the target and the object based on the detection value acquired by the sensor at the changed position; and grasping the target using at least one of the plurality of robot arms.

[0024] Regarding a method executed according to instructions stored in a non-permanent computer-readable medium, the step of grasping a target using at least one of a plurality of robot arms may include the step of grasping a target by controlling a second robot arm among the plurality of robot arms while the first robot arm grasps a sensor and detects the target.

[0025] According to a method executed according to instructions stored in a non-permanent computer-readable medium, the step of grasping a target using at least one of a plurality of robot arms may include the step of remounting the sensor to the main body by controlling the first robot arm in consideration of the state of the target and the identified object based on the detection value detected by the sensor at the changed position, and the step of grasping the target by controlling the first robot arm.

[0026] Other aspects and features of specific embodiments of the present disclosure will become more apparent from the following description taken together with the accompanying drawings, wherein the aspects and features of specific embodiments of the present disclosure are as follows:

[0027] FIG. 1 is a drawing showing a robot device according to at least one embodiment of the present disclosure.

[0028] FIG. 2 is a drawing showing a robot device according to at least one embodiment of the present disclosure.

[0029] FIG. 3 is a drawing showing a robot device including a reel cable according to at least one embodiment of the present disclosure.

[0030] FIG. 4 is a drawing showing a robot device including a neck according to at least one embodiment of the present disclosure.

[0031] FIG. 5 is a block diagram of a robot device according to at least one embodiment of the present disclosure.

[0032] FIGS. 6a, FIGS. 6b and FIGS. 6c are drawings for illustrating a robot device that changes the position and sensing direction of a sensor according to at least one embodiment of the present disclosure.

[0033] FIGS. 7a and FIGS. 7b are drawings for illustrating a method for gripping a target according to at least one embodiment of the present disclosure.

[0034] FIGS. 8 to 10 are flowcharts illustrating a method of operation of a robot device according to at least one embodiment of the present disclosure.

[0035] Hereinafter, at least one embodiment is described in more detail with reference to the attached drawings. The embodiments described in this specification may be modified in various ways. Specific embodiments may be depicted in the drawings and described in detail in the detailed description. However, specific embodiments disclosed in the attached drawings are merely intended to facilitate understanding of at least one embodiment. Accordingly, at least one embodiment of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutions of said embodiment.

[0036] The singular form of the noun corresponding to the item may include one or multiple items, unless the relevant context clearly indicates otherwise.

[0037] In this document, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0038] The term "and / or" includes a combination of multiple related described components or any of the multiple related described components.

[0039] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0040] Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0041] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this document, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0042] When it is said that a component is "connected," "combined," "supported," or "in contact" with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0043] In this disclosure, the expression "identical" means not only complete agreement but also includes differences to the extent that account for processing error ranges.

[0044] When it is said that a component is located "on" another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0045] In the following description, the same reference numerals refer to the same elements throughout the specification. Terms such as "unit," "module," "member," and "block" may be embodied in hardware or software. As used herein, a plurality of "units," "modules," "members," and "blocks" may be implemented as a single component, and a single "unit," "module," "member," and "block" may comprise a plurality of components.

[0046] Below, a robot device (1) according to various embodiments will be described in detail with reference to the drawings.

[0047] FIG. 1 is a drawing showing a robot device (1) according to at least one embodiment of the present disclosure.

[0048] In FIG. 1, the robot device (1) may include a main body (10), a plurality of robot arms (2000), a sensor (3000), and a driver (4000).

[0049] The main body (10) is configured to include devices for overall management of the operation of the robot device (1). The main body (10) is a device that forms the outer shape of the robot device (1). In FIG. 1, the main body (10) may include a robot head (1000), a robot body (1100), and a driver (4000).

[0050] The robot head (1000) may be positioned on the upper side of the robot body (1100). The robot head (1000) may include a sensor (3000). The robot head (1000) may detect the surrounding environment based on the sensing value of the sensor (3000). The sensor (3000) may be formed integrally with the robot head (1000) or may be formed in a form positioned inside the robot head (1000). The robot head (1000) may be detachably attached to the main body (10).

[0051] The robot body (1100) may be positioned on the lower side of the robot head (1000). The robot body (1100) may be positioned in the center of the robot device (1). The robot body (1100) is a device that protects various key components of the robot device (1) on the inner side. A plurality of robot arms (2000) may be positioned on both sides of the robot body (1100). The robot body (1100) may include a robot leg (1200). In FIG. 1, the robot leg (1200) is a member that connects the driver (4000) and the robot body (1100). However, it is not necessarily limited to this, and the robot leg (1200) may be formed in various shapes according to the user's intention. For example, if the manufacturer of the robot device (1) manufactures the robot device (1) as a humanoid robot, the robot leg (1200) may be formed in a shape including two legs. If the manufacturer manufactures the robot device (1) as a quadrupedal robot, the robot leg (1200) may be formed in a shape including four legs.

[0052] The driver (4000) may be positioned on the lower side of the robot body (1100). The driver (4000) can move the robot device (1) to a specific position by the power of a built-in motor. In FIG. 1, the driver (4000) is formed as a wheel, but is not necessarily limited thereto. For example, if the robot device (1) is a humanoid robot, the driver (4000) may be omitted, and driving may be performed in a walking form by sequential driving of two separated robot legs (1200). Even if the robot device (1) is a quadruped robot, the driver (4000), such as a wheel, may be omitted, and the robot legs (1200) may take over the role of the driver.

[0053] The sensor (3000) may be positioned in front of the robot head (1000). The sensor (3000) may be positioned in a detachable manner from the main body (10) or the robot head (1000). In FIG. 1, two sensors (3000) are positioned in front of the robot head (1000), but are not necessarily limited thereto and may be formed as a single sensor or as a plurality of three or more sensors.

[0054] A plurality of robot arms (2000) may be disposed on the main body (10). Specifically, a plurality of robot arms (2000) may be attached to both sides of the robot body (1100). The plurality of robot arms (2000) are configured to move each joint of a plurality of joints to move an end effector to a specific position. The plurality of robot arms (2000) may be formed integrally with the main body (10), or they may exist as modules independent of the main body (10) and be formed in a form assembled to the main body (10).

[0055] FIG. 2 is a drawing showing a robot device (1) according to at least one embodiment of the present disclosure.

[0056] In FIG. 2, a plurality of robot arms (2000) may include a first robot arm (2100) and a second robot arm (2200). The first robot arm (2100) may include a plurality of first joints (2110, 2120) and a first end effector (2130). The second robot arm (2200) may include a plurality of second joints (2210, 2220) and a second end effector (2230).

[0057] A plurality of first joints (2110, 2120) and a plurality of second joints (2210, 2220) can move in various ways around each axis. Although the plurality of first joints (2110, 2120) and a plurality of second joints (2210, 2220) are all illustrated as I-shaped modules in FIG. 2, they are not necessarily limited thereto and can be formed as modules of various shapes, such as L-shaped modules. The plurality of robot arms (2000) are not limited to the first robot arm (2100) and the second robot arm (2200), and may include three or more different robot arms.

[0058] Each of the first end effector (2130) and the second end effector (2230) can be assembled on one side of a plurality of first joints (2110, 2120) and a plurality of second joints (2210, 2220). The first end effector (2130) and the second end effector (2230) are modules that the robot device (1) interacts directly with an object or environment to perform a specific task.

[0059] Depending on the type or shape of the end effector, the robot device (1) can perform various tasks such as grasping an object, striking it, or making a hole. Various embodiments of the present disclosure are illustrated and described based on the case where the task of grasping an object is performed.

[0060] The first end effector (2130) and the second end effector (2230) may be equipped with the function of gripping a specific object. The first end effector (2130) and the second end effector (2230) may include a gripper. The first end effector (2130) and the second end effector (2230) may grip an object placed between the grippers via a drive motor and may perform various actions such as lifting the object or moving the position of the object.

[0061] The robot head (1000) may be formed integrally with the sensor (3000). The robot head (1000) may include a detachable connecting part (3100) with the sensor (3000). The connecting part (3100) is an intermediate member that can combine or separate the robot head (1000) and the main body (10). The connecting part (3100) may include a magnetic guide. Through this, the connecting part (3100) can stably combine the robot head (1000) and the main body. However, the connecting part (3100) is not limited thereto and can be implemented in various forms. For example, the connecting part (3100) may include an insertion area into which the sensor (3000) can be inserted. The sensor (3000) may be manufactured in a size and shape corresponding to the insertion area so that it can be inserted into or separated from the insertion area. Alternatively, the connecting portion (3100) may be provided with at least one groove into which at least one projection provided on the sensor (3000) can be inserted. In addition, the connecting portion (3100) and the sensor (3000) may be connected or separated in various ways.

[0062] FIG. 3 is a drawing showing a robot device (1) including a reel cable (3400) according to at least one embodiment of the present disclosure.

[0063] The robot device (1) may include a first interface (3200) and a second interface (3300). When occlusion is identified while the sensor (3000) is coupled to the robot head (1000) through the first interface (3200), the robot device (1) may control the first robot arm (2100) or the second robot arm (2200) to detach the sensor (3000) of the robot head (1000) from the first interface (3200) and mount it to the second interface (3300) provided on the first robot arm (2100) or the second robot arm (2200). In FIG. 3, the second interface (3300) is shown positioned in the center of the first end effector (2130) and the second end effector (2230), but is not necessarily limited thereto and can be positioned at various locations on the first end effector (2130) and the second end effector (2230). Accordingly, even if the robot head (1000) or the sensor (3000) is separated from the robot device (1), the sensor (3000) can operate normally through the first interface (3200) and the second interface (3300).

[0064] The robot device (1) may include a reel cable (3400). The robot head (1000) may include a reel cable (3400) connected to a sensor (3000). The reel cable (3400) may be extended according to the position of the sensor (3000) when the sensor (3000) is separated from the main body (10). Accordingly, the robot device (1) allows the sensor (3000) to operate normally through the reel cable (3400) even if the robot head (1000) or the sensor (3000) is separated.

[0065] In FIG. 3, the robot device (1) is shown with the first interface (3200), the second interface (3300), and the reel cable (3400) all combined together, but is not necessarily limited thereto, and the robot device (1) may be formed in a form including only the first interface (3200) and the second interface (3300), or in a form including only the reel cable (3400).

[0066] FIG. 4 is a drawing showing a robot device (1) including a neck (5000) according to at least one embodiment of the present disclosure.

[0067] In FIG. 4, the robot device (1) may include a neck (5000). The neck (5000) may be positioned between the main body (10) and the robot head (1000). The neck (5000) may move the robot head (1000) and the sensor (3000) to various positions through a plurality of neck joints (5010, 5020, 5030). The neck (5000) may include a plurality of neck joints (5010, 5020, 5030). The plurality of neck joints (5010, 5020, 5030) may be formed integrally with the main body (10) or may exist as modules independent of the main body (10) and be formed in a form assembled to the main body (10). The plurality of neck joints (5010, 5020, 5030) may move in various ways around each axis. Although the multiple neck joints (5010, 5020, 5030) are all shown as I-shaped modules in FIG. 4, they are not necessarily limited to this and can be formed into various types of modules, such as L-shaped modules.

[0068] FIG. 5 is a block diagram of a robot device (1) according to at least one embodiment of the present disclosure.

[0069] The robot device (1) may include a memory (6010), a processor (6020), and a plurality of robot arms (2000).

[0070] The memory (6010) is configured to contain various programs, instructions, and data required for the operation of the robot device (1). At least one instruction can be stored in the memory (6010). Although the memory (6010) is depicted as being separate from the processor (6020) in FIG. 5, it is not necessarily limited thereto, and the memory (6010) may be implemented as an internal memory such as ROM (e.g., EEPROM (electrically erasable programmable read-only memory)) or RAM included in the processor (6020).

[0071] Alternatively, the memory (6010) may be implemented in the form of a memory embedded in the robot device (1) or in the form of a memory that can be attached to and detached from the robot device (1), depending on the purpose of data storage. Specifically, the memory (6010) may be implemented in various forms such as volatile memory, SRAM (static RAM), or SDRAM (synchronous dynamic RAM), etc., non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory, hard drive, or solid state drive (SSD), CF (compact flash), SD (secure digital), MicroSD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.

[0072] In the present disclosure, the term memory (6010) may be used to include a storage unit, a ROM or RAM within a processor (6020), or a memory card (e.g., a micro SD card, a memory stick) mounted in an electronic device. Although the memory (6010) is depicted as one in FIG. 5, the memory (6010) may be implemented in various numbers.

[0073] The memory (6010) is configured to store at least one instruction, O / S (Operating System), program, and data, etc., regarding the robot device (1).

[0074] The memory (6010) is accessed by the processor (6020). In the memory (6010), reading, writing, modifying, deleting, updating, etc. of data by the processor (6020) can be performed.

[0075] Specifically, the memory (6010) may store various information such as assembly status information of each joint of a plurality of robot arms (2000), shape information of a robot head (1000), attachment position information of a first interface (3200), maximum extension distance range information of a reel cable (3400), maximum sensing range information of a sensor (3000), driving speed information of a driver (4000), assembly status information of each joint of a neck (5000), and programs and commands for controlling the operation of the robot device (1) and other devices.

[0076] The memory (6010) can store multiple artificial intelligence models that have been trained. For example, the artificial intelligence models can be implemented as CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory), DNN (Deep Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), etc., but are not limited to such examples. These artificial intelligence models are computing systems implemented based on the neural networks of human or animal brains, and may be referred to as learning models, machine learning models, neural network models, deep learning models, etc.

[0077] The processor (6020) is a component connected to each component of the robot device (1) to control the overall operation of the robot device (1). The processor (6020) can operate according to the execution of at least one instruction. The processor (6020) may be implemented as a digital signal processor (DSP) that processes digital video signals, a microprocessor, a GPU (Graphics Processing Unit), etc. However, it is not limited thereto, and may include or be defined by one or more of a central processing unit (CPU), MCU (Micro Controller Unit), MPU (micro processing unit), controller, application processor (AP), communication processor (CP), or ARM processor. Additionally, the processor (6020) may be implemented as a System on Chip (SoC) or Large Scale Integration (LSI) with a built-in processing algorithm, or may be implemented in the form of an Application Specific Integrated Circuit (ASIC) or Field Programmable Gate Array (FPGA).

[0078] The processor (6020) may perform at least one of the various operations described above based on an artificial intelligence model. The processor (6020) for executing the artificial intelligence model may be implemented through a combination of software and a general-purpose processor such as a CPU, AP, DSP (Digital Signal Processor), a graphics-dedicated processor such as a GPU, VPU (Vision Processing Unit), or an artificial intelligence-dedicated processor such as an NPU.

[0079] If the processor (6020) is implemented as a processor dedicated to artificial intelligence, it may be designed as a hardware chip such as an ASIC or FPGA specialized in processing a specific artificial intelligence model.

[0080] When the processor (6020) is implemented as a dedicated processor, it may be implemented to include a memory (6010) for implementing an embodiment of the present disclosure, or may be implemented to include a memory processing function for using external memory. The processor (6020) may be implemented as one or multiple processors. Additionally, the processor (6020) may perform various operations individually or collectively based on programs, instructions, data, etc. stored in the memory (6010).

[0081] The processor (6020) can identify a target based on the sensing value of the sensor (3000) and perform operations on the target using at least one of the plurality of robot arms (2000). The processor (6020) can identify whether the target is occluded by an object.

[0082] Occlusion can be identified in various ways. For example, if the sensor (3000) includes a camera, the processor (6020) can identify a target and an object in a captured image taken by the camera, and identify whether there is occlusion based on whether there is overlap and the overlap ratio, etc. Specifically, the processor (6020) divides all pixels in the captured image into multiple pixel groups and then extracts the pixel representative values ​​of the pixels included in each pixel group. The processor (6020) identifies the location of pixel groups having pixel representative values ​​that are within a similar range, and if multiple similar pixel groups are located consecutively, it identifies that the similar pixel groups form an edge of a single object. The processor (6020) can estimate what kind of object it is based on the shape of the edge, the pixel values ​​of the pixels belonging within the edge, etc., and identify the distance to the object based on the size of the edge.

[0083] When the target object to be grasped, i.e., the target, is an apple, the processor (6020) identifies an edge forming a round closed loop within the captured image, and if the pixel values ​​included in the closed loop are a color corresponding to the apple (e.g., red), the target can be identified as being fully sensed. On the other hand, if part of the target is obscured by an object, part of the edge identified within the captured image is deformed. That is, a part of the round closed loop becomes a shape that is compressed inward. The processor (6020) can determine that occlusion has occurred if the ratio between the size of the entire edge and the size of the compressed part is greater than a certain ratio (e.g., 10%). The above description of the object identification method and the occlusion identification method is merely an example and is not limited thereto.

[0084] When the processor (6020) identifies that occlusion has occurred, it can control the first robot arm (2100) among the plurality of robot arms (2000) to grasp the sensor (3000) of the main body (10) and detach it from the main body (10) to change the position and sensing direction of the sensor (3000). Here, although it has been described that the processor (6020) controls the first robot arm (2100) among the plurality of robot arms (2000), it is not necessarily limited thereto, and the processor (6020) can control the second robot arm (2200) among the plurality of robot arms (2000) to grasp and detach the sensor (3000) of the main body (10) to change the position and sensing direction of the sensor (3000). In the following description, the processor (6020) is to be described based on the premise that it controls the first robot arm (2100) among the plurality of robot arms (2000), but the second robot arm (2200) among the plurality of robot arms (2000) will also be capable of the same control operation as the first robot arm (2100). In addition, the control operation of the first robot arm (2100) and the second robot arm (2200) of the plurality of robot arms (2000) may include the control operation of the plurality of first joints (2110, 2120), the plurality of second joints (2210, 2220), the first end effector (2130), and the second end effector (2230).

[0085] Afterwards, the processor (6020) can control the first robot arm (2100) among the plurality of robot arms (2000) to identify the state of the target and object based on the sensing value sensed by the sensor (3000) at the changed position.

[0086] When the state of the target and the object is identified based on the sensing value sensed by the sensor (3000) at the changed position, the processor (6020) can control the second robot arm (2200), which is another of the plurality of robot arms (2000), to grasp the target. Subsequently, when the state of the target and the object is identified based on the sensing value sensed by the sensor (3000) at the changed position, the processor (6020) can mount the sensor (3000) on the main body (10) and control the first robot arm (2100) to grasp the target. Subsequently, when interference with the object occurs during the process of grasping the target, the processor (6020) can use at least one of the plurality of robot arms (2000) to first grasp and move the object, and then grasp the target. In the present disclosure, interference occurs not only when the robot arm or the end effector of the robot arm directly contacts an object during the target grasping process, but also when the probability of contacting the object is greater than a preset probability. That is, interference occurs even before contact with the object, including cases where it is identified that contact with the object will occur if the grasping operation is continued further.

[0087] In the event that interference occurs, if the second robot arm (2200) among the multiple robot arms (2000) is grasping the sensor (3000) and changing its position, the processor (6020) can control the first robot arm (2100), which is the other of the multiple robot arms (2000), to grasp the target. That is, the processor (6020) can control the first robot arm (2100) and the second robot arm (2200) among the multiple robot arms (2000) in opposite directions to each other.

[0088] When occlusion is identified while the sensor (3000) is connected to the robot head (1000) through the first interface (3200), the processor (6020) can control the first robot arm (2100) to detach the sensor (3000) of the robot head (1000) from the first interface (3200) and mount it to the second interface (3300) provided on the first robot arm (2100).

[0089] The processor (6020) can control the first robot arm (2100) to sequentially move the sensor (3000) to multiple positions facing the target from different directions relative to the target. A position for sensing the target means a position facing the target. For example, if the sensor (3000) is a camera, the processor (6020) can move the camera near the target and then adjust the direction so that the camera lens faces the target.

[0090] The processor (6020) can identify the characteristics of the target and the characteristics of the target object by combining the total sensing values ​​sensed at multiple locations and directions.

[0091] In the above-described part, the processor (6020) can control the first robot arm (2100) and the second robot arm (2200) respectively. At this time, the tasks that the first robot arm (2100) can perform can also be performed by the second robot arm (2200), and the tasks that the second robot arm (2200) performs can also be performed by the first robot arm (2100). The processor (6020) can control the first robot arm (2100) and the second robot arm (2200) so that the order of the control operations of the first robot arm (2100) and the control operations of the second robot arm (2200) among the plurality of robot arms (2000) are swapped.

[0092] FIGS. 6a, FIGS. 6b and FIGS. 6c are drawings for explaining a robot device (1) that changes the position and sensing direction of a sensor (3000) according to at least one embodiment of the present disclosure.

[0093] In FIG. 6a, the robot device (1) can identify the placement status of a target (8400) and objects (8100, 8200, 8300, 8500) using a sensor (3000) placed on the robot head (1000) from the front. The robot device (1) can identify the placement status of the target (8400) and objects (8100, 8200, 8300, 8500) within a maximum sensing radius range (7100). The robot device (1) can identify whether there is an occlusion where the target (8400) is obscured by an object (8100, 8200, 8300, 8500) based on the sensing value of the sensor (3000) placed on the robot head (1000). Here, occlusion refers to a situation where the target (8400) is obscured by or comes into contact with an object (8100, 8200, 8300, 8500), thereby interfering with the operation of the robot device (1). The robot device (1) can estimate the area obscured by the visible shape of the target (8400) based on the sensing value of the sensor (3000). Subsequently, the robot device (1) can determine the location of the sensor (3000) for occlusion resolution.

[0094] In FIG. 6b, the robot device (1) can change the position and sensing direction of the robot head (1000) or sensor (3000) by controlling the first robot arm (2100) among the plurality of robot arms (2000) to grasp and detach the robot head (1000) or sensor (3000) of the main body (10). Likewise, in FIG. 6c, the robot device (1) can change the position and sensing direction of the robot head (1000) or sensor (3000) by controlling the second robot arm (2200) among the plurality of robot arms (2000) to grasp and detach the robot head (1000) or sensor (3000) of the main body (10).

[0095] In FIGS. 6b and 6c, the robot device (1) can identify the state of the target (8400) and objects (8100, 8200, 8300, 8500) based on the sensing value sensed by the robot head (1000) or sensor (3000) at the changed position. The robot device (1) can identify the placement state of the target (8400) and objects (8100, 8200, 8300, 8500) within the maximum sensing radius range (7200, 7300). That is, the robot device (1) can obtain data on the state of the target (8400) and objects (8100, 8200, 8300, 8500) by separating and moving the robot head (1000) itself or the sensor (3000) mounted on the robot head (1000) through one of the plurality of robot arms (2000).

[0096] Specifically, in FIG. 6b, the robot device (1) can control the first robot arm (2100) among a plurality of robot arms (2000) to grasp and detach the robot head (1000) or sensor (3000) of the main body (10) and position the robot head (1000) or sensor (3000) on the left side of the target (8400) and objects (8100, 8200, 8300, 8500). The robot device (1) can change the sensing direction of the robot head (1000) or sensor (3000) positioned on the left side of the target (8400) and objects (8100, 8200, 8300, 8500) so that it faces the target (8400) and objects (8100, 8200, 8300, 8500). Afterwards, the robot device (1) can identify the state of the target (8400) and objects (8100, 8200, 8300, 8500) based on the sensing value of the sensor (3000).

[0097] Likewise, in FIG. 6c, the robot device (1) can control the second robot arm (2200) among the plurality of robot arms (2000) to grasp and detach the robot head (1000) or sensor (3000) of the main body (10) and position the robot head (1000) or sensor (3000) on the right side of the target (8400) and objects (8100, 8200, 8300, 8500). The robot device (1) can change the sensing direction of the robot head (1000) or sensor (3000) positioned on the right side of the target (8400) and objects (8100, 8200, 8300, 8500) so that it faces the target (8400) and objects (8100, 8200, 8300, 8500). Subsequently, the robot device (1) can identify the state of the target (8400) and the object (8100, 8200, 8300, 8500) based on the sensing value of the sensor (3000). However, it is not necessarily limited to this, and the robot device (1) can move the robot head (1000) or the sensor (3000) to a position where various states of the target (8400) and the object (8100, 8200, 8300, 8500) can be identified after separating the robot head (1000) or the sensor (3000) through one of the plurality of robot arms (2000), and change the sensing direction of the robot head (1000) or the sensor (3000) toward the target (8400) and the object (8100, 8200, 8300, 8500).

[0098] In FIG. 6a, when the target (8400) is viewed from the front, occlusion occurs due to at least one location of the objects (8100, 8200, 8300, 8500). Thus, the robot device (1) can identify that the target (8400) is obscured by the objects (8100, 8200, 8300, 8500). Specifically, the robot device can identify that the target (8400) is obscured by the fifth object (8500). In FIG. 6b, the robot head (1000) or sensor (3000) senses the target (8400) and the objects (8100, 8200, 8300, 8500) from the left side. When viewed from the left side, occlusion where the target (8400) is obscured by the first object (8100) can be identified. In FIG. 6c, the robot head (1000) or sensor (3000) senses the target (8400) and objects (8100, 8200, 8300, 8500) from the right side. When viewed from the right side, occlusion where the target (8400) is obscured by objects is not identified. The robot device (1) can synthesize these data to determine the state of the target (8400) and objects (8100, 8200, 8300, 8500), the position of the gripping point of the target (8400), whether there is an interfering object on the gripping path of the target (8400), and whether there is movement after gripping the interfering object.

[0099] FIGS. 7a and FIGS. 7b are drawings for illustrating a method for grasping a target (8400) according to at least one embodiment of the present disclosure.

[0100] In FIG. 7a, the robot device (1) can identify the placement status of a target (8400) and objects (8100, 8200, 8300, 8500) using a sensor (3000) placed on the robot head (1000) from the front. The robot device (1) can identify the placement status of the target (8400) and objects (8100, 8200, 8300, 8500) within a maximum sensing radius range (7100). The robot device (1) can identify whether there is an occlusion where the target (8400) is obscured by an object (8100, 8200, 8300, 8500) based on the sensing value of the sensor (3000) placed on the robot head (1000). Here, occlusion refers to a situation where the target (8400) is obscured by or comes into contact with an object (8100, 8200, 8300, 8500), thereby interfering with the operation of the robot device (1). The robot device (1) can estimate the area obscured by the visible shape of the target (8400) based on the sensing value of the sensor (3000). Subsequently, the robot device (1) can determine the location of the sensor (3000) for occlusion resolution.

[0101] In FIG. 7a, the robot device (1) can change the position and sensing direction of the robot head (1000) or sensor (3000) by controlling the first robot arm (2100) among the plurality of robot arms (2000) to grasp and detach the robot head (1000) or sensor (3000) of the main body (10). Here, controlling the first robot arm (2100) means the same as controlling the plurality of first joints (2110, 2120) and the first gripper (2130). Likewise, the robot device (1) can change the position and sensing direction of the robot head (1000) or sensor (3000) by controlling the second robot arm (2200) among the plurality of robot arms (2000) to grasp and detach the robot head (1000) or sensor (3000) of the main body (10). Here, controlling the second robot arm (2200) means the same as controlling multiple second joints (2210, 2220) and the second gripper (2130).

[0102] The robot device (1) can identify the state of a target (8400) and an object (8100, 8200, 8300, 8500) based on a sensing value sensed from a robot head (1000) or a sensor (3000) at a changed position. That is, the robot device (1) can obtain data on the state of the target (8400) and an object (8100, 8200, 8300, 8500) by separating and moving the robot head (1000) itself or the sensor (3000) mounted on the robot head (1000) through one of the plurality of robot arms (2000). Subsequently, the robot device (1) can grasp and move the target (8400) and objects (8100, 8200, 8300, 8500) through another of the plurality of robot arms (2000) while performing an operation to identify the state of the target (8400) and objects (8100, 8200, 8300, 8500) at the robot head (1000) or sensor (3000) at the changed position. Specifically, the robot device (1) can grasp and move the fifth object (8500), which is an interfering object, and grasp the target (8400) through another of the plurality of robot arms (2000) while performing an operation to identify the state of the target (8400) and objects (8100, 8200, 8300, 8500) at the robot head (1000) or sensor (3000) at the changed position.

[0103] In FIG. 7b, the robot device (1) can change the position and sensing direction of the robot head (1000) or sensor (3000) by controlling the first robot arm (2100) among the plurality of robot arms (2000) to grasp and detach the robot head (1000) or sensor (3000) of the main body (10). Likewise, the robot device (1) can change the position and sensing direction of the robot head (1000) or sensor (3000) by controlling the second robot arm (2200) among the plurality of robot arms (2000) to grasp and detach the robot head (1000) or sensor (3000) of the main body (10).

[0104] The robot device (1) can identify the state of a target (8400) and an object (8100, 8200, 8300, 8500) based on a sensing value sensed by a robot head (1000) or a sensor (3000) at a changed position. That is, the robot device (1) can obtain data on the state of the target (8400) and the object (8100, 8200, 8300, 8500) by separating and moving the robot head (1000) itself or the sensor (3000) mounted on the robot head (1000) through one of the multiple robot arms (2000). Afterward, the robot device (1) can return the robot head (1000) or the sensor (3000) at the changed position back to its original position. Subsequently, the robot device (1) can control the first robot arm (2100) and the second robot arm (2200) of the plurality of robot arms (2000) to grasp and move the target (8400) and the object (8100, 8200, 8300, 8500). That is, when the state of the target (8400) and the object (8100, 8200, 8300, 8500) is identified based on the sensing value sensed by the sensor (3000) at the changed position, the robot device (1) can mount the sensor (3000) on the main body (10) and control the first robot arm (2100) or the second robot arm (2200) to grasp the target (8400). Here, the first robot arm (2100) and the second robot arm (2200) have the same meaning as controlling each of the plurality of first joints (2110, 2120), the plurality of second joints (2210, 2220), the first gripper (2130), and the second gripper (2230). If the robot device (1) identifies that interference with an object (8100, 8200, 8300, 8500) will occur during the process of grasping a target (8400), it may first grasp and move the object (8100, 8200, 8300, 8500) using at least one of the plurality of robot arms (2000), and then grasp the target (8400).In FIG. 7b, the robot device (1) can grasp and move the first object (8100) and the fifth object (8500) using the first robot arm (2100) and the second robot arm (2200) of the plurality of robot arms (2000), and then grasp the target (8400) with one of the plurality of robot arms (2000).

[0105] FIG. 8 is a flowchart showing a method of operation of a robot device (1) according to at least one embodiment of the present disclosure.

[0106] In FIG. 8, when the robot device (1) identifies an occlusion in which the target (8400) is obscured by an object (8100, 8200, 8300, 8500) based on the sensing value of the sensor (3000), it can control the first robot arm (2100) among the plurality of robot arms (2000) to grasp the sensor (3000) and separate it from the main body (10) (S1010). Subsequently, the robot device (1) can identify the state of the target (8400) and the object (8100, 8200, 8300, 8500) based on the sensing value sensed by the sensor (3000) at the changed position (S1020). Subsequently, the robot device (1) can grasp the target (8400) using at least one of the plurality of robot arms (2000) (S1030).

[0107] In the step of grasping a target (8400) using at least one of a plurality of robot arms (2000), the robot device (1) can grasp the target (8400) by controlling a second robot arm (2200), which is another of the plurality of robot arms (2000), while the first robot arm (2100) senses the target (8400).

[0108] Here, detailed descriptions of the target (8400), multiple robot arms (2000), sensors (3000), etc. have been provided above, so a description thereof is omitted.

[0109] FIG. 9 is a flowchart showing a method of operation of a robot device (1) according to at least one embodiment of the present disclosure.

[0110] The robot device (1) can identify the state of the target (8400) and the object (8100, 8200, 8300, 8500) based on the sensing value sensed by the sensor (3000) at the changed position (S1110). Afterward, when the state of the target (8400) and the object (8100, 8200, 8300, 8500) is identified based on the sensing value sensed by the sensor (3000) at the changed position, the robot device (1) can control the first robot arm (2100) to remount the sensor (3000) to the main body (10) (S1120). Afterward, the robot device (1) can control the first robot arm (2100) to grasp the target (8400) (S1130).

[0111] Here, if it is identified that interference with the object (8100, 8200, 8300, 8500) will occur during the process of grasping the target (8400), the robot device (1) can first grasp and move the object (8100, 8200, 8300, 8500) using at least one of the plurality of robot arms (2000), and then grasp the target (8400).

[0112] Here, detailed descriptions of the target (8400) and objects (8100, 8200, 8300, 8500) have been provided above, so such descriptions are omitted.

[0113] FIG. 10 is a flowchart showing a method of operation of a robot device (1) according to at least one embodiment of the present disclosure.

[0114] The robot device (1) can identify whether the target (8400) is occluded based on the sensing value of the sensor (3000) (S1201). Subsequently, the robot device (1) can estimate the occluded area of ​​the target (8400) based on the visible shape based on the sensing value of the sensor (3000) (S1202). Subsequently, the robot device (1) can determine the position of the sensor (3000) for solving the occlusion where the target (8400) is obscured (S1203). The robot device (1) can obtain data regarding the state of the target (8400) and objects (8100, 8200, 8300, 8500) by separating and moving the robot head (1000) itself or the sensor (3000) mounted on the robot head (1000) through one of the plurality of robot arms (2000) (S1204). Subsequently, the robot device (1) can determine the overall shape of the unobstructed target (8400) and object (8100, 8200, 8300, 8500) by combining the sensing value of the sensor (3000) at the changed position with data regarding the state of the target (8400) and object (8100, 8200, 8300, 8500) obtained from the front (S1205). Subsequently, the robot device (1) can determine whether the occlusion of the target (8400) can be resolved based on the obtained data (S1206). Here, if the occlusion of the target (8400) cannot be resolved based on the obtained data, the robot device (1) can start again from the step of determining the position of the sensor (3000) for resolving the occlusion where the target (8400) is obscured. If the occlusion of the target (8400) can be resolved based on the secured data, the robot device (1) can determine the optimal gripping point according to the overall shape of the target (8400) (S1207).

[0115] Subsequently, the robot device (1) can check the operating range of at least one of the multiple robot arms (2000) for grasping the target (8400) (S1208). Subsequently, the robot device (1) can determine whether it is possible to grasp the target (8400) without interference from the objects (8100, 8200, 8300, 8500) (S1209). Here, if it is possible to grasp the target (8400) without interference from the objects (8100, 8200, 8300, 8500), the robot device (1) can grasp the target (8400). If it is impossible to grasp the target (8400) without interference from the objects (8100, 8200, 8300, 8500), the robot device (1) can distinguish the objects (8100, 8200, 8300, 8500) on the target (8400) grasping path based on the acquired data (S1210). Subsequently, the robot device (1) can control at least one of the plurality of robot arms (2000) to grasp the objects (8100, 8200, 8300, 8500) and move them to a location other than the target (8400) grasping path (S1211). Subsequently, the robot device (1) can grasp the target (8400) (S1212).

[0116] Each of the components described in this document may consist of one or more components, and the names of such components may vary depending on the type of electronic device.

[0117] Although at least one embodiment of the present disclosure has been described individually above, each embodiment is not necessarily required to be implemented alone, and the configuration and operation of each embodiment may be implemented in combination with at least one other embodiment.

[0118] Although preferred embodiments have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In a robot device, entity; A plurality of robot arms mounted on the above main body; A sensor connected to the above main body in a detachable manner; Memory in which at least one instruction is stored; and It includes at least one processor configured to execute the above at least one instruction; and The above at least one instruction is, When executed individually or collectively by at least one processor, the robot device, Identifying a target based on the sensing value of the sensor, and performing a task related to the target using at least one of the plurality of robot arms, If it is detected that the above target is obscured by an object, Control the first robot arm among the plurality of robot arms to grasp the sensor and detach it from the main body to change the position and sensing direction of the sensor, and A robot device that identifies the state of the target and the object based on the sensing value of the sensor acquired at a changed location.

2. In Paragraph 1, The above at least one instruction is, When executed individually or collectively by at least one processor, the robot device, A robot device that controls a second robot arm among the plurality of robot arms to grasp the target when the state of the target and the object is identified.

3. In Paragraph 1, The above at least one instruction is, When executed individually or collectively by at least one processor, the robot device, A robot device that, when the state of the target and the object is identified, controls the first robot arm to mount the sensor on the main body and grasp the target.

4. In Paragraph 2, The above at least one instruction is, When executed individually or collectively by at least one processor, the robot device, A robot device that, when interference caused by the object occurs during the process of grasping the target, uses at least one of the plurality of robot arms to grasp and move the object before grasping the target.

5. In Paragraph 1, The above main body is, Robot body; A driver positioned on the lower side of the above-mentioned robot body; and A robot head mounted on the upper side of the robot body; comprising The plurality of robot arms are coupled to both sides of the robot body, and The above sensor is a robot device mounted on the robot head.

6. In Paragraph 5, The robot head above comprises: a connection part configured to be connected to the sensor; and It further includes a reel cable connected to the above sensor, and The above reel cable is a robot device that can be extended based on the state in which the sensor is separated from the main body.

7. In Paragraph 5, The robot head includes a first interface configured to be connected to the sensor, and The end effector of at least one first robot arm among the plurality of robot arms includes a second interface configured to be connected to the sensor, The above at least one instruction is, When executed individually or collectively by at least one processor, the robot device, A robot device that, when the occlusion is identified while the sensor is coupled to the robot head through the first interface, controls the first robot arm to detach the sensor from the first interface and mount it to the second interface provided on the first robot arm.

8. In Paragraph 1, The above at least one instruction is, When executed individually or collectively by at least one processor, the robot device, A robot device that controls the first robot arm to sequentially move the sensor to a plurality of positions facing the target, and combines the sensing values ​​obtained from the plurality of positions to identify the characteristics of the target and the characteristics of the target object.

9. In a method for controlling a robot device, When it is identified that the target is obscured by an object, the first robot arm among the plurality of robot arms of the robot device is controlled to grasp the sensor of the robot device and separate it from the main body of the robot device; A step of moving the first robot arm to change the position and sensing direction of the sensor; A step of identifying the state of the target and the object based on the sensing value obtained from the sensor at the changed location; and A method for controlling a robot device comprising the step of grasping a target using at least one of the plurality of robot arms.

10. In Paragraph 9, The step of grasping the target using at least one of the plurality of robot arms is, A method for controlling a robot device, comprising the step of controlling a second robot arm among a plurality of robot arms to grasp a target while the sensor grasped by the first robot arm senses the target.

11. In Paragraph 9, The step of grasping the target using at least one of the plurality of robot arms is, When the state of the target and the object is identified based on the sensing value sensed by the sensor at the changed position, the step of controlling the first robot arm to remount the sensor to the main body; and A method for controlling a robot device, comprising the step of controlling the first robot arm to grasp the target.

12. In Paragraph 11, A control method further comprising the step of, if interference with the object occurs during the process of grasping the target, first grasping and moving the object using at least one of the plurality of robot arms before grasping the target.

13. In Paragraph 9, The step of controlling the first robot arm among the plurality of robot arms to grasp the sensor and separate it from the main body is A method for controlling a robot device, comprising the step of, when the occlusion is identified while the sensor is connected to a first interface provided in the robot head of the robot device, controlling the first robot arm to disconnect the sensor from the first interface mounted in the robot head of the robot device and connecting it to a second interface provided in the end effector of the first robot arm.

14. In Paragraph 9, The step of moving the first robot arm to change the position and sensing direction of the sensor is, A method for controlling a robot device, comprising the step of controlling the first robot arm to sequentially move the sensor to a plurality of positions facing the target.

15. In Paragraph 14, The step of identifying the state of the above target and the above object is, A control method for a robot device comprising the step of identifying the characteristics of the target and the characteristics of the object by combining all sensing values ​​sensed at the plurality of positions and directions.

Images