Method for extracting objects and electronic device supporting same

Abstract

An electronic device according to various embodiments comprises: a camera; a robot arm; a memory for storing instructions; and at least one processor, wherein the processor may be configured to: acquire a first image by photographing a plurality of objects by using the camera; identify the shape and arrangement of the plurality of objects by using the first image; determine an extraction order of the plurality of objects on the basis of the shape and arrangement of the plurality of objects; determine an extraction posture of the robot arm with respect to the objects on the basis of the shape and arrangement of the plurality of objects; initiate extraction of the plurality of objects by controlling the robot arm on the basis of the extraction order and the extraction posture of the robot arm; while extraction of the plurality of objects is being performed, determine whether a designated condition for photographing at least one remaining object among the plurality of objects is satisfied, by using a sensor disposed on the robot arm; if it is determined that the designated condition is not satisfied, extract the at least one remaining object on the basis of the first image without photographing the at least one remaining object; if it is determined that the designated condition is satisfied, acquire a second image by photographing the at least one remaining object; and extract the at least one remaining object on the basis of the second image.

Description

Method for extracting objects and electronic device supporting the same

[0001] Various embodiments according to the present disclosure relate to a method for extracting objects and an electronic device supporting the same.

[0002] Robot technology is being widely utilized in the manufacturing and logistics industries due to the increasing demand for process automation. In particular, there is a growing need to handle small components or sensitive materials accurately and quickly during the production and assembly processes of electronic devices. To this end, extraction technology using robotic arms plays a crucial role, contributing to improved production efficiency, minimizing errors caused by manual labor, and enhancing safety in the work environment. Specifically, the technology for extracting irregularly stacked materials or objects is advancing through the convergence of robotic arm control systems and computer vision technology.

[0003] However, existing robotic arm extraction technology is primarily focused on handling small-sized objects. This is because it has been designed to efficiently process small electronic components, logistics boxes, or lightweight industrial materials. Yet, in tasks involving medium or large objects (e.g., large panels, heavy objects, large machine parts, etc.), it is often difficult to achieve efficient and stable extraction using only existing technology.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0005] When an electronic device extracts and loads small, medium, or large objects, problems may arise, such as the difficulty in extracting and transporting the objects, the difficulty in calculating deviation amounts using gyroscope values, and the long time required to photograph and recognize the objects.

[0006] Accordingly, various embodiments of the present invention are intended to provide a method and apparatus for efficiently extracting objects using an electronic device.

[0007] The technical problems to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description in this disclosure.

[0008] An electronic device according to one embodiment of the present invention may include a robot arm, a memory for storing instructions, and at least one processor. When instructions are executed, the processor may acquire a first image by photographing a plurality of objects using a camera. When instructions are executed, the processor may identify the shape and arrangement of the plurality of objects using the first image. When instructions are executed, the processor may determine the extraction order of the plurality of objects based on the shape and arrangement of the plurality of objects. When instructions are executed, the processor may determine the extraction posture of the robot arm for the objects based on the shape and arrangement of the plurality of objects. When instructions are executed, the processor may initiate extraction of the plurality of objects by controlling the robot arm based on the extraction order and the extraction posture of the robot arm. When instructions are executed, while extraction of the plurality of objects is being performed, the processor may determine whether a specified condition for photographing at least one remaining object among the plurality of objects is satisfied using a sensor placed on the robot arm. When the processor determines that a specified condition is not satisfied when the instructions are executed, it may not photograph at least one remaining object and may extract at least one remaining object based on the first image. When the processor determines that a specified condition is satisfied when the instructions are executed, it may acquire a second image by photographing at least one remaining object and extract at least one remaining object based on the second image.

[0009] FIG. 1 is a drawing for illustrating an example of an electronic device extracting objects according to one embodiment of the present disclosure.

[0010] FIG. 2 is a flowchart illustrating a process in which an electronic device extracts objects according to one embodiment of the present disclosure.

[0011] FIG. 3 is a drawing for explaining an example in which an electronic device, according to one embodiment of the present disclosure, photographs a plurality of objects and identifies the shape and arrangement of the plurality of objects.

[0012] FIG. 4 is a drawing for explaining an example in which an electronic device determines the extraction order of a plurality of objects according to one embodiment of the present disclosure.

[0013] FIG. 5 is a drawing for explaining an example in which an electronic device determines the extraction order and extraction posture of a plurality of objects according to one embodiment of the present disclosure.

[0014] FIG. 6 is a drawing for explaining an example in which an electronic device determines the extraction posture of a plurality of objects according to one embodiment of the present disclosure.

[0015] FIG. 7 is a flowchart illustrating a process in which an electronic device selects a plurality of objects from a plurality of objects according to one embodiment of the present disclosure.

[0016] FIG. 8 is a drawing for explaining an example of an electronic device extracting a plurality of objects from a mobile loading device loaded with a plurality of objects, according to one embodiment of the present disclosure.

[0017] FIG. 9 is a drawing for explaining an example of an electronic device extracting a plurality of objects from a mobile loading device loaded with a plurality of objects, according to one embodiment of the present disclosure.

[0018] FIG. 10 is a block diagram of an electronic device according to various embodiments of the present disclosure.

[0019] FIG. 11 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0020] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0021] The technical problems to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this invention belongs from the description below.

[0022] Hereinafter, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0023] The terms used in this disclosure are described in their current, general form considering the functions mentioned herein; however, they may refer to various other terms depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Accordingly, the terms used in this disclosure should not be interpreted solely by their names, but should be interpreted based on the meaning of the terms and the overall content of this disclosure.

[0024] Additionally, terms such as "first," "second," etc., may be used to describe various components, but the components are not limited by these terms. These terms are used for the purpose of distinguishing one component from another.

[0025] Phrases such as "in one embodiment" appearing in various places in this disclosure do not necessarily refer to the same embodiment.

[0026] In this disclosure, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In actual devices, connections between components may be represented by various alternative or additional functional connections, physical connections, or circuit connections.

[0027] In the present disclosure, a plurality of objects may be objects that are to be extracted by an electronic device. For example, a plurality of objects may be materials extracted by a robotic arm of an electronic device.

[0028] In the present disclosure, a plurality of objects may be loaded on a movable loading device, and the plurality of objects may include a plurality of objects that are subject to extraction by an electronic device and objects that are not subject to extraction by an electronic device.

[0029] FIG. 1 is a drawing for explaining an example in which an electronic device (100) extracts objects according to one embodiment of the present disclosure.

[0030] According to one embodiment, a plurality of objects may be loaded onto a mobile loading device, and an electronic device (100) may remove the plurality of objects from the mobile loading device using a robot arm. For example, referring to FIG. 1, as shown in identification number 110, three objects (111, 112, 113) are loaded onto a mobile loading device (115), and an electronic device (100) may remove the three objects (111, 112, 113) from the mobile loading device using a robot arm (104).

[0031] According to one embodiment, an electronic device (100) can extract multiple objects from a movable loading device based on a single image in which multiple objects are captured. The electronic device (100) can obtain a first image by capturing multiple objects and extract multiple objects using only the first image. The first image may be an image in which the electronic device (100) captures multiple objects before initiating extraction of the multiple objects. For example, referring to FIG. 1, the electronic device (100) can obtain a first image by capturing multiple objects (111, 112, 113) before initiating extraction according to identification number 110, and can extract three objects (111, 112, 113) from a movable loading device based on the first image, which is a single image. Even when extracting the second object (112) according to identification number 130, the electronic device (100) can extract the second object (112) using the first image without separate shooting.

[0032] According to one embodiment, the electronic device (100) can perform extraction of a plurality of objects without re-photographing based on a first image. The electronic device (100) according to the present disclosure can perform extraction of a plurality of objects without re-photographing based on a first image, which is a single image, and can omit the process of extracting objects by photographing and setting the posture of the robot arm each time extraction is performed for each of the plurality of objects. Through this, the electronic device (100) according to the present disclosure can achieve the effect of enabling efficient operation when extracting objects and enabling rapid extraction of objects.

[0033] According to one embodiment, the electronic device (100) identifies the shape and arrangement of a plurality of objects from a first image in which a plurality of objects are captured, determines the extraction order of the plurality of objects and the extraction posture of the robot arm, and can extract the plurality of objects according to the determined extraction order. For example, before starting extraction according to identification number 110, the electronic device (100) determines the extraction order for a plurality of objects (111, 112, 113) and the extraction posture of the robot arm (104), and can start extraction based on the determined extraction order and the extraction posture of the robot arm (104). Accordingly, when extracting a second object (112) according to identification number 130, the extraction posture of the robot arm (104) for the second object (112) may be the extraction posture of the robot arm (104) according to the extraction posture determined by the electronic device (100) before starting extraction.

[0034] According to one embodiment, while extracting a plurality of objects, a second image may be captured only when it is determined that extraction of a specific object is impossible using a sensor (105) placed on a robot arm, and the remaining objects may be extracted based on the second image. For example, when an electronic device (100) extracts a second object (112) as in identification number 130, if it is determined that extraction of the second object (112) is impossible using a sensor (105) placed on a robot arm (104), a second image may be captured for two objects (112, 113) excluding the first object (111) that has already been extracted. In this case, the electronic device (100) may extract the two objects (112, 113) based on the second image.

[0035] FIG. 2 is a flowchart illustrating a process in which an electronic device (100) extracts objects according to one embodiment of the present disclosure.

[0036] In operation 210, the electronic device (100) can obtain a first image by photographing a plurality of objects.

[0037] According to one embodiment, the electronic device (100) can acquire a first image by photographing a plurality of objects at a distance where the plurality of objects can be included in a single image. The electronic device (100) can acquire a first image by photographing at a position moved to be spaced apart from the plurality of objects by a certain distance. For example, the electronic device (100) can move based on the field of view of the camera of the electronic device (100) to photograph all of the plurality of objects. Alternatively, based on the field of view of the camera of the electronic device (100), a movable loading device containing the plurality of objects may be moved so that all of the plurality of objects are included in the first image.

[0038] According to one embodiment, a plurality of objects may be loaded on a mobile loading device. For example, the plurality of objects may be loaded in a stacked manner from bottom to top while lying down. For example, the plurality of objects may be loaded vertically while standing upright. For example, the plurality of objects may be loaded in a manner where each object is fixed individually by a fixing tool of the mobile loading device. For example, the plurality of objects may be loaded by overlapping or stacking with other objects.

[0039] According to one embodiment, a plurality of objects may be a term referring to materials extracted by a robot arm of an electronic device. For example, a plurality of objects may be a term including large materials that are the subject of extraction. In this case, large materials may refer to materials having an area of ​​at least 1500mm x 800mm on one side, for example, when considering a rectangular material. However, the plurality of objects according to the present disclosure are not limited to the large materials described above and may also refer to small materials.

[0040] According to one embodiment, an electronic device (100) can obtain a first image by photographing a plurality of objects using a depth camera. For example, the electronic device (100) can obtain a first image containing depth data for a plurality of objects by measuring depth data for a plurality of objects through a depth camera. For example, the electronic device (100) can obtain a first image containing RGB information and depth information for a plurality of objects through a depth camera.

[0041] According to one embodiment, the first image may include depth information for a plurality of objects. For example, the first image may include depth values, 3D coordinates, RGB-D information, and information regarding the slope and curvature of the surfaces of the plurality of objects. For example, the first image may include information regarding a depth map for a plurality of objects. However, the depth information is not limited to the examples described above, and the first image may include 3D spatial information for a plurality of objects.

[0042] In operation 220, the electronic device (100) can identify the shape and arrangement of a plurality of objects using the first image.

[0043] According to one embodiment, an electronic device (100) can identify the shape and arrangement of a plurality of objects based on depth information of a first image. For example, the electronic device (100) can identify a 3D shape by extracting the outlines and surfaces of a plurality of objects based on depth information of a plurality of objects included in a first image. Additionally, the electronic device (100) can identify the arrangement in space by calculating the position and distance between the electronic device (100) and a plurality of objects, or between a plurality of objects, based on depth information included in the first image. Furthermore, the electronic device (100) can separate each object by utilizing the depth difference between objects included in the depth information of the first image, and can identify a plurality of objects and the background behind the plurality of objects.

[0044] According to one embodiment, the electronic device (100) can identify the arrangement of multiple objects by using a first image to determine how far the electronic device (100) is from the multiple objects. For example, the electronic device (100) can identify multiple objects using a segmentation algorithm along with depth information of the multiple objects, and can calculate the distance from the electronic device (100) by identifying the center point of each object. Through this, the electronic device (100) can identify the arrangement of multiple objects.

[0045] According to one embodiment, an electronic device (100) can identify the shape and arrangement of a plurality of objects using a first image captured by a depth camera and software. For example, the electronic device (100) can obtain data in other formats that the electronic device (100) can use by processing data regarding the arrangement and shape of a plurality of objects included in the first image using a 3D design tool and a modeling tool. The electronic device (100) can identify the shape and arrangement of a plurality of objects through the processed data. More specifically, for example, the electronic device (100) can identify the shape and arrangement of a plurality of objects using a first image and CAD (computer-aided design) software.

[0046] According to one embodiment, an electronic device (100) can identify the shape and arrangement of a plurality of objects based on six degrees of freedom (6 DOF) using a first image. The electronic device (100) can identify the position of a plurality of objects based on the X, Y, and Z axes and the degree of rotation of a plurality of objects based on the X, Y, and Z axes using the first image. The electronic device (100) can identify the shape and arrangement of a plurality of objects using the position and degree of rotation of a plurality of objects based on the X, Y, and Z axes. For example, the electronic device (100) can identify the shape and arrangement of a plurality of objects by using a pose estimation method that accurately estimates the position and orientation of an object or an object recognition method that identifies an object and tracks its position and rotation state.

[0047] According to one embodiment, an electronic device (100) can identify the shape and arrangement of a plurality of objects using product information regarding a plurality of objects based on a first image. The product information may include information regarding the shape of the plurality of objects. The electronic device (100) can extract product information from memory. For example, the electronic device (100) can extract product information regarding the shape of a plurality of objects from memory. The electronic device (100) can determine whether the first image was accurately captured by identifying the shape and arrangement of a plurality of objects in the first image using information regarding a plurality of objects included in the first image and product information. The electronic device (100) can determine whether the shape of the plurality of objects is correct and can identify and determine the arrangement of the plurality of objects using information regarding a plurality of objects included in the first image and product information. Furthermore, the electronic device (100) can select a plurality of objects to be extracted from among the plurality of objects by comparing the product information with the information included in the first image, which will be described later in FIG. 7.

[0048] In operation 230, the electronic device (100) can determine the extraction order of a plurality of objects based on the first image.

[0049] In one embodiment, the electronic device (100) can determine the extraction order of a plurality of objects according to a user's settings. The criteria for the extraction order of the plurality of objects may be pre-set by the user. For example, the electronic device (100) can determine the extraction order for a plurality of objects identified according to the criteria set by the user. More specifically, the user may set the priority to be given, for example, starting with the objects placed on the left, and the electronic device (100) can determine to extract the objects placed on the left first among the plurality of objects according to the user's settings.

[0050] In one embodiment, the electronic device (100) may determine the extraction order based on the arrangement of a plurality of objects. For example, the electronic device (100) may determine the extraction order such that, among the plurality of objects, objects arranged closer to the electronic device (100) are extracted first. Alternatively, for example, the electronic device (100) may determine the extraction order such that objects arranged to the right of the electronic device (100) are extracted first. Alternatively, for example, the electronic device (100) may determine the extraction order such that, among the plurality of objects stacked in a stacked structure, objects stacked on top are extracted first. However, the above examples are not limited, and the electronic device (100) may determine the extraction order of the plurality of objects in other ways.

[0051] In one embodiment, the electronic device (100) may determine the extraction order based on the size of a plurality of objects. For example, the electronic device (100) may determine the extraction order such that, among the plurality of objects, the object with the largest area identified in the first image is extracted first. For example, the electronic device (100) may determine the extraction order such that the object having a width close to the width of the robot arm (104) is extracted first. As another example, the electronic device (100) may determine the extraction order such that the object with the highest degree of correspondence to the shape of the plurality of objects according to product information is extracted first. However, the above-described examples are not limited, and the electronic device (100) may determine the extraction order of the plurality of objects in other ways.

[0052] In one embodiment, the electronic device (100) can determine a score for a plurality of objects based on the shape and arrangement of the plurality of objects and determine the extraction order of the plurality of objects based on the score. In one embodiment, the electronic device (100) can determine a score by comparing information included in a first image with product information extracted from memory and determine the extraction order so that the object with the higher score among the plurality of objects is extracted first. For example, the electronic device (100) can compare information regarding the shape of the plurality of objects included in the first image with product information regarding the shape of the plurality of objects to be extracted, determine that an object that accurately corresponds to the product information in relation to size, length, or area has a high score, and determine the extraction order of the plurality of objects based on the score. In one embodiment, the electronic device (100) can determine a score for the objects based on the extraction posture of the robot arm (104) for the plurality of objects and determine the extraction order of the plurality of objects based on the determined score. For example, the electronic device (100) can determine a score for each extraction position of the robot arm (104) through items such as stability and posture conformity, and determine the extraction order based on the score. The details of determining the extraction order by determining the score based on the extraction position of the robot arm (104) will be further examined in operation 240. However, examples of determining scores for multiple objects and determining the extraction order based on the score are not limited to the examples described above.

[0053] In operation 240, the electronic device (100) can determine the extraction posture of the robot arm for a plurality of objects based on the first image.

[0054] According to one embodiment, the electronic device (100) can identify the shape and arrangement of a plurality of objects using a first image and determine the extraction posture of the robot arm (104) based on the shape and arrangement of the plurality of objects. The electronic device (100) can determine the extraction posture of the robot arm (104) through 3D coordinate values ​​and rotation values ​​related to the shape and arrangement of the plurality of objects. The electronic device (100) can determine the extraction posture of the robot arm (104) by using the geometric characteristics of the plurality of objects included in product information related to the shape of the plurality of objects. For example, the electronic device (100) can determine the optimal posture in which the robot arm (104) can grasp the plurality of objects according to the shape and arrangement of the plurality of objects.

[0055] According to one embodiment, the electronic device (100) can determine the extraction position of the robot arm (104) for all of the multiple objects based on a first image in which multiple objects are captured. The electronic device (100) can determine the extraction position of the robot arm (104) for each of the multiple objects included in the first image. Furthermore, even if there are objects among the multiple objects that are obscured by other objects, the electronic device (100) can determine the extraction position of the robot arm (104) for the obscured objects by using product information related to the shapes of the multiple objects. For example, if there is an object among the multiple objects that is obscured by another object, the electronic device (100) can determine the extraction position of the robot arm (104) for the obscured object by predicting the shape of the obscured object using product information regarding the shapes of the objects.

[0056] According to one embodiment, the electronic device (100) can determine the extraction posture of the robot arm (104) by using an algorithm based on the shape and arrangement of a plurality of objects identified based on a first image. The electronic device (100) can determine the extraction posture of the robot arm (104) by analyzing the first image using an algorithm based on at least one of ICP (Iterative Closest Point), REGTR (End-to-end Point Cloud Correspondences with Transformers), or FPFH (Fast Point Feature Histogram). For example, the electronic device (100) can determine the extraction posture of the robot arm (104) by using an ICP-based algorithm to register two point clouds (reference point cloud and input point cloud) and calculate a transformation matrix. For example, the electronic device (100) can determine the extraction posture of the robot arm (104) by using a REGTR-based algorithm to learn the correspondence between two point clouds through a transformation architecture and perform alignment. For example, the electronic device (100) can determine the extraction posture of the robot arm (104) by using an FPFH-based algorithm to calculate the position and posture of objects by comparing feature points.

[0057] According to one embodiment, the electronic device (100) can determine the extraction posture of the robot arm (104) for a plurality of objects and determine a score for each object according to the extraction posture of the robot arm (104). For example, the electronic device (100) can determine a score for each object by comparing the shapes of the plurality of objects identified using product information regarding the shapes of the plurality of objects and a first image, such that a higher matching rate results in a higher score. For example, in the extraction posture of the robot arm (104) determined for each object, the object requiring the least movement of the robot arm (104) can be determined to have a high score. For example, in the extraction posture of the robot arm (104) determined for each object, the object requiring the least change in posture of the robot arm (104) can be determined to have a high score. For example, in the extraction position of the robot arm (104) determined for each object, the stability when the object is grasped according to the extraction position of the robot arm (104) can be determined so that the object with higher stability has a higher score.

[0058] According to one embodiment, as described above in operation 230, the electronic device (100) can determine the extraction position of the robot arm (104) for a plurality of objects, determine a score for each object based on the extraction position of the robot arm (104), and determine the extraction order so that the objects with the higher scores are extracted first. That is, in one embodiment, the electronic device (100) can determine the extraction position of the robot arm (104) for a plurality of objects in operation 240 and determine the extraction order of the plurality of objects according to operation 230.

[0059] In one embodiment, operation 230 and operation 240 may be performed in parallel by the electronic device (100) and may also be performed simultaneously. Additionally, operation 240 may be performed before operation 230, and the order of performance is not limited to that of the drawings.

[0060] In operation 250, the electronic device (100) can initiate extraction of a plurality of objects based on the first image.

[0061] According to one embodiment, the electronic device (100) can initiate extraction of a plurality of objects by controlling the robot arm (104) based on the extraction order and the extraction posture of the robot arm (104). That is, the electronic device (100) can initiate extraction of a plurality of objects sequentially according to the extraction order and the extraction posture of the robot arm (104) that have already been determined prior to the initiation of extraction. In particular, for example, if the plurality of objects are large in size and heavy in weight, the shape and arrangement of the objects will be maintained even if the robot arm (104) extracts the objects; therefore, the electronic device (100) can stably initiate extraction of a plurality of objects according to the extraction order and the extraction posture of the robot arm (104) that have been determined prior to the initiation of extraction.

[0062] According to one embodiment, the electronic device (100) can initiate extraction of a plurality of objects based on a first image, which is a single image. The electronic device (100) determines the extraction order of the plurality of objects and the extraction posture of the robot arm (104) for the plurality of objects based on the first image, which is a single image, and can initiate extraction of a plurality of objects without the need to additionally determine the extraction order or the extraction posture of the robot arm (104). The electronic device (100) initiates extraction of a plurality of objects based on the first image, and since the objects will be smoothly extracted based on the extraction posture of the robot arm (104), extraction of a plurality of objects based on the first image can be initiated without additionally using a camera to take pictures in the event of any abnormalities.

[0063] According to one embodiment, the electronic device (100) can initiate extraction of a plurality of objects based on the shape and arrangement of the plurality of objects identified based on a first image taken before extracting the plurality of objects, without the need to monitor through a camera while extracting the plurality of objects. For example, even when the electronic device (100) extracts object A, which is the first extraction order, and then extracts object B, which is the second extraction order, the electronic device (100) extracts object B based on the first image taken before initiating extraction, so the electronic device (100) can initiate extraction using a robot arm (104) without monitoring object B through a camera.

[0064] According to one embodiment, an electronic device (100) can initiate extraction of a plurality of objects using a robot arm (104) with six degrees of freedom (6 DOF). The electronic device (100) can initiate extraction of a plurality of objects by using the robot arm (104) to perform linear movement along the X, Y, and Z axes and rotational movement around the X, Y, and Z axes. For example, the robot arm (104) of the electronic device (100) can perform linear movement through X, Y, and Z movement, and can initiate extraction of a plurality of objects by performing rotational movement around each axis through Roll, Pitch, and Yaw rotation.

[0065] In operation 260, the electronic device (100) can determine whether a specified condition is satisfied.

[0066] According to one embodiment, the electronic device (100) can determine whether a specified re-shooting condition is satisfied based on sensing information generated from a sensor (105) placed on a robot arm (104). The electronic device (100) can determine whether a specified re-shooting condition is satisfied based on sensing information generated from at least one of a pressure sensor or a position sensor placed on the robot arm (104). However, the sensor (105) placed on the robot arm (104) is not limited to the example described above.

[0067] According to one embodiment, the electronic device (100) may determine that a specified re-shooting condition is satisfied when extraction of a plurality of objects cannot be performed according to the extraction order of a plurality of objects determined based on a first image or the extraction posture of the robot arm (104) for the plurality of objects. For example, the electronic device (100) may determine that extraction of a plurality of objects cannot be performed when the object to be extracted according to the extraction order of a plurality of objects is not recognized using a sensor placed on the robot arm (104). For example, the electronic device (100) may determine that extraction of a plurality of objects cannot be performed when the robot arm (104) fails to grasp an object according to the extraction posture of the robot arm (104) for the plurality of objects.

[0068] According to one embodiment, the electronic device (100) can determine whether a specified re-shooting condition is satisfied by identifying whether the robot arm (104) has grasped at least one remaining object according to the extraction position of the robot arm (104). The electronic device (100) can determine that the specified re-shooting condition is satisfied if the robot arm (104) has not grasped at least one remaining object according to the determined extraction position of the robot arm (104). The electronic device (100) can determine whether the robot arm (104) has grasped an object through at least one of a pressure sensor or a position sensor placed on the robot arm (104). For example, if the robot arm (104) of the electronic device (100) must grasp an object based on the determined extraction position of the robot arm (104), but the object is not recognized by the sensor (105) of the robot arm (104), the electronic device (100) can determine that the robot arm (104) has not grasped an object.

[0069] According to one embodiment, the case in which the robot arm (104) fails to grasp at least one remaining object may include the case in which the robot arm (104) fails to completely grasp the object according to the determined extraction posture of the robot arm (104). For example, if the object slips and falls from the robot arm (104) during the process of the robot arm (104) grasping the object, the electronic device (100) may determine that the robot arm (104) failed to grasp the object. For example, if the robot arm (104) fails to grasp the part of the object it is supposed to grasp and grasps a different part, the electronic device (100) may identify this through sensing information via a sensor (105) placed on the robot arm (104) and determine that the robot arm (104) failed to grasp the object. For example, if the weight sensed by the robot arm (104) while extracting an object differs from the weight according to product information for multiple objects, the electronic device (100) may determine that the robot arm (104) failed to grasp the object.

[0070] According to one embodiment, the electronic device (100) may perform operation 270 when it determines that a specified re-shooting condition is satisfied. According to one embodiment, the electronic device (100) may perform operation 290 when it determines that a specified re-shooting condition is not satisfied.

[0071] According to one embodiment, if the electronic device (100) determines that the specified re-shooting condition is not satisfied, it performs operation 290 so that all of the multiple objects can be extracted based on "one" first image.

[0072] In operation 290, the electronic device (100) can extract at least one remaining object based on the first image.

[0073] According to one embodiment, if the electronic device (100) determines that the specified re-shooting conditions are not satisfied, it may extract at least one remaining object based on the first image without photographing at least one remaining object. If the electronic device (100) determines that the specified re-shooting conditions are not satisfied, it may extract objects sequentially based on the first image without re-shooting the remaining objects.

[0074] According to one embodiment, if there is no problem with the electronic device (100) extracting based on the extraction posture of the robot arm (104) according to the extraction sequence, it can continue to extract the remaining objects based on the first image.

[0075] According to one embodiment, since a plurality of objects are extracted based on a first image, the remaining objects can be extracted at once based on the first image without the need to take a new image.

[0076] In operation 270, the electronic device (100) can obtain a second image by photographing at least one remaining object.

[0077] According to one embodiment, the electronic device (100) may acquire a second image only when it is determined that a specified re-shooting condition is satisfied. If the specified re-shooting condition is not satisfied, the electronic device (100) may not take a second image and may extract all of the multiple objects based on a single first image. That is, the electronic device (100) takes a second image for the remaining objects only when it is unable to extract all of the multiple objects based on the first image.

[0078] According to one embodiment, the electronic device (100) can obtain a second image by photographing at least one remaining object, which is the remainder excluding the objects already extracted from a plurality of objects.

[0079] According to one embodiment, the electronic device (100) can obtain a second image containing depth data by photographing at least one remaining object using a depth camera.

[0080] According to one embodiment, the electronic device (100) can move considering the camera's field of view to capture at least one remaining object in one image.

[0081] In one embodiment, operation 270 may be an operation corresponding to the operation of operation 210.

[0082] In operation 280, the electronic device (100) can extract at least one remaining object based on the second image.

[0083] According to one embodiment, the electronic device (100) can identify the shape and arrangement of at least one remaining object using a second image. In one embodiment, compared to operation 220, there is a difference in that the electronic device (100) identifies the shape and arrangement of the object using a second image rather than a first image, and otherwise the electronic device (100) can perform an operation corresponding to operation 220.

[0084] According to one embodiment, the electronic device (100) can determine the extraction order of at least one remaining object based on a second image. In one embodiment, there is a difference from operation 230 in that the electronic device (100) determines the extraction order of the object based on a second image rather than a first image, and otherwise the electronic device (100) can perform an operation corresponding to operation 230.

[0085] According to one embodiment, the electronic device (100) can determine the extraction position of the robot arm (104) for at least one remaining object based on the second image. In one embodiment, compared with operation 240, there is a difference in that the electronic device (100) determines the extraction position of the robot arm (104) for the object based on the second image rather than the first image, and otherwise the electronic device (100) can perform an operation corresponding to operation 240.

[0086] According to one embodiment, the electronic device (100) can extract at least one remaining object based on a second image. In one embodiment, compared to operation 250, there is a difference in that the electronic device (100) initiates extraction of the object based on a second image rather than a first image, and otherwise the electronic device (100) can perform an operation corresponding to operation 250.

[0087] According to one embodiment, the electronic device (100) can determine whether a specified condition is satisfied while extracting at least one remaining object based on a second image. In one embodiment, compared with operation 260, there is a difference in that the electronic device (100) determines whether a specified re-shooting condition is satisfied while initiating extraction of the object based on a second image rather than a first image, and otherwise the electronic device (100) can perform an operation corresponding to operation 260.

[0088] In one embodiment, when the electronic device (100) initiates extraction of at least one remaining object based on a second image, the electronic device (100) can extract all of at least one remaining object based on the second image, which is one image for at least one remaining object.

[0089] FIG. 3 is a drawing for explaining an example in which an electronic device (100) according to one embodiment of the present disclosure photographs a plurality of objects and identifies the shape and arrangement of the plurality of objects.

[0090] Referring to FIG. 3, according to one embodiment, an electronic device (100) can photograph a plurality of objects using a camera capable of measuring depth data. For example, the electronic device (100) can photograph a plurality of objects using a depth camera. However, the type of camera according to the present disclosure is not limited to a depth camera, and other cameras capable of obtaining depth information, such as a stereo camera, mono vision, LIDAR (light detection and ranging), infrared camera, and ultrasonic camera, may be used.

[0091] According to one embodiment, the electronic device (100) can move to photograph a plurality of objects in one frame. For example, referring to identification number 310, the electronic device (100) can move to photograph a plurality of objects (111, 112, 113) all in one frame using a camera. The electronic device (100) can move to photograph a plurality of objects in one frame according to the camera's field of view (FOV) or field of view.

[0092] The electronic device (100) according to the present disclosure can acquire a first image by taking a single shot of a plurality of objects. However, depending on the situation, the first image may be acquired by taking more than one shot of a plurality of objects. According to one embodiment, although not illustrated, the electronic device (100) may take multiple shots from multiple angles to identify a plurality of objects included in a single frame. Specifically, the electronic device (100) can determine the shape and arrangement of the overlapping parts by taking shots from different angles of objects that are arranged in an overlapping manner and whose shape and arrangement cannot be determined by taking only one shot. In this case, even if the shots are taken two or more times, the electronic device (100) acquires a first image, which is a single image formed by combining the results captured by the camera.

[0093] According to one embodiment, the electronic device (100) can identify the shape and arrangement of a plurality of objects through a first image. For example, the electronic device (100) can photograph a plurality of objects (111, 112, 113) according to identification number 310 and acquire a first image (400) to identify the shape and arrangement of the plurality of objects (111, 112, 113). Referring to the first image (400) in FIG. 3, the electronic device (100) can identify an arrangement in which three objects (401, 402, 403) are arranged, and can identify the shape of the objects through the first image (400) shown in FIG. 3.

[0094] The three objects (111, 112, 113) of identification number 310 in FIG. 3 may correspond to the three objects (401, 402, 403) included in the first image (400). The electronic device (100) may photograph the three objects (111, 112, 113) of identification number 310 to obtain the first image (400) containing the three objects (401, 402, 403).

[0095] According to one embodiment, the electronic device (100) can identify the shape and arrangement of a plurality of objects using shape data of a plurality of objects included in a first image. For example, referring to FIG. 3, the electronic device (100) can identify the shape and arrangement of a plurality of objects by separating the plurality of objects from the background using shape data of a plurality of objects included in a first image (400) captured using a depth camera.

[0096] According to one embodiment, an electronic device (100) can identify the shape and arrangement of a plurality of objects using product information regarding a plurality of objects. The electronic device (100) can identify the shape and arrangement of a plurality of objects in a first image using a first image of a plurality of objects and product information regarding the shape of the plurality of objects. For example, the electronic device (100) can identify the shape and arrangement of a plurality of objects from a first image (400) using product information regarding the plurality of objects with respect to a first image of a plurality of objects.

[0097] According to one embodiment, the first image may include an image resulting from the segmentation of objects based on depth data obtained by the electronic device (100) photographing a plurality of objects. For example, the first image according to identification number 400 of FIG. 3 may be an image resulting from the separation of objects from the background based on depth data obtained by the electronic device (100) photographing a plurality of objects at identification number 310.

[0098] According to one embodiment, the first image may include position coordinate values ​​and / or rotation values ​​for a plurality of objects. For example, the first image according to identification number 400 of FIG. 3 may include position coordinate values ​​and / or rotation values ​​of a plurality of objects identified based on depth data obtained by the electronic device (100) at identification number 310 photographing a plurality of objects.

[0099] FIG. 4 is a drawing for explaining an example in which an electronic device (100) determines the extraction order of a plurality of objects according to one embodiment of the present disclosure.

[0100] The identification number 400 shown in FIG. 4 may be a drawing showing a first image containing a plurality of objects. The first image (400) shown in FIG. 4 may be the same image as the first image according to the identification number 400 of FIG. 3.

[0101] The image of identification number 410 may be the first image of identification number 400. For convenience of explanation, the image of identification number 410 shows the object (401) to be extracted first in the first image of identification number 400 in a distinguishable manner.

[0102] The image of identification number 420 may be the first image of identification number 400. For convenience of explanation, the image of identification number 420 shows the object (402) to be extracted as the second priority from the first image of identification number 400 in a distinguishable manner.

[0103] The image of identification number 430 may be the first image of identification number 400. For convenience of explanation, the image of identification number 430 shows the object (403) to be extracted as the third priority from the first image of identification number 400 in a distinguishable manner.

[0104] In one embodiment, the electronic device (100) can determine the extraction order of a plurality of objects based on a first image. For example, the electronic device (100) may decide to extract the object placed on the right first, according to the user's settings. Accordingly, the electronic device (100) may determine the extraction order of the object (401) indicated by identification number 410 as the first priority. The electronic device (100) may determine the extraction order of the object (402) indicated by identification number 420 as the second priority. The electronic device (100) may determine the extraction order of the object (403) indicated by identification number 430 as the third priority.

[0105] In one embodiment, since the electronic device (100) determines the extraction order of the plurality of objects before initiating extraction of the plurality of objects, when the electronic device (100) determines the extraction order, the objects of higher priority may also be displayed together on the first image. For example, in the first image of identification number 420, in addition to the object (402) to be extracted in the second priority, the object to be extracted in the first priority may also be displayed together, and in the first image of identification number 430, in addition to the object (403) to be extracted in the third priority, the objects to be extracted in the first and second priority may also be displayed together.

[0106] According to one embodiment, the method by which the electronic device (100) determines the extraction order of a plurality of objects may vary depending on the user's settings, determined score, etc., and may be determined according to the above-mentioned content.

[0107] According to one embodiment, examples for determining the first image and the extraction order are not limited to the examples described above, and images taken in a manner different from the examples in the drawings may be used, and are not limited to the examples according to the present disclosure.

[0108] FIG. 5 is a drawing for explaining an example in which an electronic device (100) determines the extraction order and extraction posture of a plurality of objects according to one embodiment of the present disclosure.

[0109] The images of identification numbers 410, 420, and 430 shown in FIG. 5 may be the first images of identification number 400 in FIG. 4. For convenience of explanation, the images of identification numbers 410, 420, and 430 in FIG. 5 are images in which objects (401, 402, 403) to be extracted according to the extraction order are distinguished from the first images of identification number 400 in FIG. 4.

[0110] Identification numbers 511, 521, and 531 are data processed from the first image (400) and may be data used for the extraction operation of the robot arm (104). Identification numbers 511, 521, and 531 may be data having a format that the robot arm (104) can use by processing data regarding the arrangement and shape of a plurality of objects included in the first image (400). Identification numbers 511, 521, and 531 may be CAD (computer-aided design) data converted from the first image (400).

[0111] According to one embodiment, the electronic device (100) can identify the shape and arrangement of a plurality of objects included in a first image and determine the extraction position of the robot arm (104). The electronic device (100) can determine the extraction position of the plurality of objects using processed data obtained from the first image. For example, the electronic device (100) can obtain processed data from the first image, such as identification number 511, using the first image of identification number 410. The processed data according to identification number 511 may be CAD data. Identification number 511 may be a drawing that performs an extraction simulation of an object (401) placed on the right using the processed data. The electronic device (100) can determine the extraction position of the robot arm (104) through an extraction simulation using CAD data according to identification number 511. For example, the electronic device (100) can obtain processed data from the first image, such as identification number 521, by using the first image of identification number 420. The processed data according to identification number 521 may be CAD data. Identification number 521 may be a drawing that performs a simulation of extraction of an object (402) placed in the center using the processed data. The electronic device (100) can determine the extraction posture of the robot arm (104) through an extraction simulation using the CAD data according to identification number 521. For example, the electronic device (100) can obtain processed data from the first image, such as identification number 531, by using the first image of identification number 430. The processed data according to identification number 531 may be CAD data. Identification number 531 may be a drawing that performs a simulation of extraction of an object (403) placed in the center using the processed data. The electronic device (100) can determine the extraction posture of the robot arm (104) through an extraction simulation using CAD data according to identification number 531.

[0112] FIG. 6 is a drawing for explaining an example in which an electronic device (100) determines the extraction posture of a plurality of objects according to one embodiment of the present disclosure.

[0113] Identification numbers 611, 621, and 631 are data processed from the first image (400) and may be data used for the extraction operation of the robot arm (104). Identification numbers 611, 621, and 631 may be data having a format that the robot arm (104) can use by processing data regarding the arrangement and shape of a plurality of objects included in the first image (400). Identification numbers 611, 621, and 631 may be CAD (computer-aided design) data converted from the first image (400).

[0114] According to one embodiment, the electronic device (100) can identify the shape and arrangement of a plurality of objects included in a first image and determine the extraction posture of the robot arm. The electronic device (100) can determine the extraction posture of the plurality of objects using processed data obtained from the first image. For example, the processed data according to identification numbers 611, 621, and 631 may be CAD data. The electronic device (100) can determine the extraction posture of the robot arm through an extraction simulation using the CAD data according to identification numbers 611, 621, and 631.

[0115] According to one embodiment, the electronic device (100) can determine the extraction posture of a robot arm by considering the extraction order of a plurality of objects. The electronic device (100) can determine the extraction posture of a robot arm by considering the extraction order using processed data obtained from a first image. The electronic device (100) can determine the extraction posture by excluding objects of higher priority according to the extraction order in an extraction simulation using CAD data. The electronic device (100) can determine the extraction posture of a robot arm by assuming that objects of higher priority are not extracted in the extraction simulation. For example, identification number 611 may be a drawing that performs an extraction simulation of an object (401) of the first priority extraction order placed on the right using processed data. At this time, in the extraction simulation according to identification number 611, an object (402) of the second priority extraction order placed in the center and an object (403) of the third priority extraction order placed on the left may be included together. The electronic device (100) can determine the extraction posture of the robot arm for the object (401) with the first priority extraction order through an extraction simulation using CAD data according to identification number 611. For example, identification number 621 may be a drawing that performs an extraction simulation of an object (402) with the second priority extraction order placed in the center using processed data. In this case, the extraction simulation according to identification number 621 may include an object (403) with the third priority extraction order placed on the left, and may not include an object (401) with the first priority extraction order. The electronic device (100) can determine the extraction posture of the robot arm for the object (402) with the second priority extraction order, excluding the object (401) with the first priority extraction order, through an extraction simulation using CAD data according to identification number 621. For example, identification number 631 may be a drawing that performs a simulation of the extraction of an object (403) placed in the center with a third-rank extraction order using processed data.At this time, in the extraction simulation according to identification number 631, the object (401) with the first priority in extraction order and the object (402) with the second priority in extraction order may not be included. The electronic device (100) can determine the extraction posture of the robot arm for the object (403) with the third priority in extraction order by excluding the object (401) with the first priority in extraction order and the object (402) with the second priority in extraction order through an extraction simulation using CAD data according to identification number 631.

[0116] According to one embodiment, by determining the extraction posture of a robot arm by considering the extraction order of a plurality of objects, the effect of stably extracting an object during the extraction process can be achieved. Furthermore, as the diversity of the extraction posture of the robot arm increases, the electronic device can determine an extraction posture that allows for more stable extraction of an object among various extraction postures of the robot arm. However, the method of determining the extraction posture of the robot arm is not limited to the examples related to the CAD data or simulation described above.

[0117] FIG. 7 is a flowchart illustrating a process (700) in which an electronic device (100) selects a plurality of objects among a plurality of objects according to one embodiment of the present disclosure.

[0118] In one embodiment, a plurality of objects may be a term including objects loaded on a movable loading device and subject to extraction by an electronic device. A plurality of objects may be a term including a plurality of objects. More specifically, a plurality of objects may be a term including both objects subject to extraction by the electronic device and objects not subject to extraction by the electronic device. However, the plurality of objects in the present disclosure are not limited to the meanings described above.

[0119] According to one embodiment, the electronic device (100) can select a plurality of objects to be extracted from among a plurality of objects loaded on a movable loading device. The electronic device (100) can select a plurality of objects from among the plurality of objects using product information regarding the shapes of the plurality of objects. That is, the electronic device (100) can select a plurality of objects to be extracted from among a plurality of objects loaded on a movable loading device in order to prevent unnecessary shooting, scanning, or movement of the robot arm.

[0120] In operation 710, the electronic device (100) can extract product information from memory.

[0121] In one embodiment, the product information may include information regarding the shapes of a plurality of objects. For example, the product information may include information related to the size, shape, cross-sectional area, or weight of the plurality of objects.

[0122] In one embodiment, the product information may include information regarding the serial numbers of a plurality of objects. For example, the product information may include the serial numbers of a plurality of objects. The electronic device (100) may determine whether an object is to be extracted through the serial number.

[0123] In one embodiment, the electronic device (100) can store product information in memory. The electronic device (100) can receive product information from an external electronic device (100) or a user and store it in memory.

[0124] In operation 720, the electronic device (100) can obtain a first image by photographing a plurality of objects including a plurality of objects.

[0125] In one embodiment, the electronic device (100) may obtain a first image by photographing a plurality of objects loaded on a movable loading device. The plurality of objects may include a plurality of target objects. The first image may include a plurality of target objects and other objects that are not the target of extraction.

[0126] In operation 730, the electronic device (100) can select a plurality of objects using product information and a first image.

[0127] In one embodiment, the electronic device (100) can select a plurality of objects among a plurality of objects included in a first image. The electronic device (100) can distinguish a plurality of objects that are to be extracted. The electronic device (100) selects a plurality of objects among a plurality of objects included in the first image and can exclude other objects from the extraction target, excluding the plurality of objects from the first image. In one embodiment, the electronic device (100) can select a plurality of objects that are to be extracted by identifying the shape and arrangement of a plurality of objects included in the first image.

[0128] In one embodiment, the electronic device (100) can select multiple objects from a first image in which multiple objects are captured using product information. For example, the multiple objects can be selected by comparing information regarding the shapes of multiple objects included in the product information with information regarding the shapes of multiple objects captured and identified in the first image. If the shape identified in the first image corresponds to the shape according to the product information, the electronic device (100) can determine it as an extraction target and recognize and select it as multiple objects.

[0129] In one embodiment, after performing operation 730, the electronic device (100) can determine the extraction order of a plurality of objects based on the first image according to operation 230.

[0130] FIG. 8 is a drawing for explaining an example in which an electronic device (100) according to one embodiment of the present disclosure extracts a plurality of objects from a mobile loading device loaded with a plurality of objects. Specifically, FIG. 8 is a drawing for explaining an example in which an electronic device (100) according to one embodiment extracts a plurality of objects based on a first image without additional shooting.

[0131] According to one embodiment, a plurality of objects may be loaded in a movable loading device. The movable loading device may not be loaded with objects other than those intended for extraction by the electronic device (100), and may only be loaded with objects that are the extraction targets of the electronic device (100). For example, referring to identification number 810, the movable loading device (811) may only be loaded with a plurality of objects that are the extraction targets of the electronic device (100).

[0132] According to one embodiment, in a movable loading device, some of the objects among the plurality of objects that the electronic device (100) intends to extract may not obstruct other objects. For example, referring to identification number 810 in FIG. 8, six objects in the movable loading device (811) may be loaded spaced apart from each other by a predetermined range and arranged so as not to overlap each other.

[0133] According to one embodiment, when a plurality of objects loaded on a mobile loading device are all included in a plurality of objects to be extracted by the electronic device (100), and the plurality of objects are arranged so that they can be photographed by the camera of the electronic device (100) without overlapping, the electronic device (100) can determine that it can extract all of the plurality of objects using a single image. For example, according to identification number 810, all of the objects loaded on the mobile loading device (811) may be included in the objects to be extracted by the electronic device (100). In addition, since the plurality of objects are arranged so that they can be photographed without overlapping, the electronic device (100) can extract all of the plurality of objects without re-photographing based on a first image of the mobile loading device (811) in the state according to identification number 810.

[0134] According to one embodiment, the electronic device (100) determines whether the re-shooting condition is satisfied using a sensor (105) placed on a robot arm while extracting a plurality of objects, but if the plurality of objects are stably positioned and the electronic device (100) can extract the plurality of objects based on the extraction order and the extraction posture of the robot arm, all of the plurality of objects can be extracted without re-shooting. For example, for the plurality of objects according to identification number 810, extraction can be performed sequentially according to the extraction order and the extraction posture of the robot arm determined by the electronic device (100). Ultimately, the electronic device (100) can extract the plurality of objects sequentially based on the first image, and since the re-shooting condition will not be satisfied, all of the plurality of objects can be extracted without re-shooting, as in the state according to identification number 820.

[0135] According to one embodiment, the electronic device (100) can photograph a plurality of objects to obtain a first image, use the first image to identify the shape and arrangement of the plurality of objects, determine the extraction order of the plurality of objects and the extraction posture of the robot arm, and initiate extraction. For example, referring to identification number 810 in FIG. 8, the electronic device (100) can photograph six objects to obtain a first image, determine the extraction order according to a setting to extract from right to left, and determine the extraction posture of the robot arm for the six objects. The electronic device (100) can extract the objects without additional photography of the six objects included in the mobile loading device (811) by controlling the robot arm based on the extraction order and the extraction posture of the robot arm.

[0136] According to one embodiment, if the electronic device (100) determines that a specified condition for photographing at least one remaining object among a plurality of objects is not satisfied, it may extract at least one remaining object based on a first image. For example, based on a first image taken of six objects of identification number 810 in FIG. 8, the electronic device (100) may initiate the extraction of objects. While performing the extraction of the six objects, the electronic device (100) may determine that a specified condition is not satisfied. Accordingly, objects may be extracted based on the first image while all objects are being extracted from a movable loading device (821), such as identification number 820 in FIG. 8.

[0137] As illustrated in FIG. 8, if the electronic device (100) determines that a specified condition is not satisfied, it can extract objects based on a first image, and can complete the extraction of multiple objects based on the first image taken before initiating the extraction of multiple objects. That is, if the electronic device (100) determines that a specified condition is not satisfied, it can complete the extraction of multiple objects based on the first image taken initially without the need to perform additional shooting to obtain a second image, and thereby the effect of the electronic device (100) being able to extract objects efficiently and quickly can be achieved.

[0138] FIG. 9 is a drawing for explaining an example of an electronic device extracting a plurality of objects from a mobile loading device loaded with a plurality of objects, according to one embodiment of the present disclosure.

[0139] Referring to FIG. 9, according to one embodiment, a movable loading device may have a plurality of objects loaded thereon. The movable loading device may have a plurality of objects that the electronic device (100) intends to extract, and other objects that are not objects that the electronic device (100) intends to extract. For example, referring to identification number 910, the movable loading device may have an object (931) that is not an object that the electronic device (100) intends to extract loaded thereon. For example, referring to identification number 910 of FIG. 9, the movable loading device may have a plurality of objects (911, 912, 913, 921, 922) that are objects that the electronic device (100) intends to extract loaded thereon.

[0140] According to one embodiment, in a movable loading device, among a plurality of objects that the electronic device (100) intends to extract, some of the objects that the electronic device (100) intends to extract may obscure other objects. For example, referring to identification number 910 in FIG. 9, in the movable loading device, object C (913) may be positioned in front of object D (921) and positioned to obscure object D (921).

[0141] According to one embodiment, some of the objects that the electronic device (100) intends to extract may be rotated and arranged in the movable loading device. For example, referring to identification number 910 in FIG. 9, object E (922) in the movable loading device may not be arranged upright from the perspective of the electronic device (100) but may be arranged in a rotated manner.

[0142] According to one embodiment, the electronic device (100) can obtain a first image by photographing a plurality of objects loaded on a mobile loading device. For example, referring to identification number 910, the electronic device (100) can obtain a first image by photographing a plurality of objects (911, 912, 913, 921, 922, 931) loaded on a mobile loading device.

[0143] According to one embodiment, an electronic device (100) can select a plurality of objects among a plurality of objects loaded on a movable loading device using product information and a first image. The electronic device (100) can extract product information regarding the shape of the plurality of objects from memory. For example, by referring to identification number 910, the electronic device (100) can extract product information regarding the shape of the plurality of objects, and the product information may include information regarding the size of the plurality of objects, information regarding the material, information regarding the serial number, information regarding the shape of the cross-section, etc. Through the product information, the electronic device (100) can select a plurality of objects that are the target of extraction by the electronic device (100) among a plurality of objects included in the first image. The electronic device (100) can select the remaining objects excluding another object F (931) that does not correspond to the product information. The electronic device (100) can select a plurality of objects (911, 912, 913, 921, 922).

[0144] According to one embodiment, the electronic device (100) can determine the extraction order of a plurality of objects based on the shape and arrangement of the plurality of objects. For example, referring to identification number 910, the electronic device (100) can determine the extraction order of object A (911), which is placed on the far left, as the first priority. The electronic device (100) can determine the extraction order of object B (912), which is placed upright, as the second priority. The electronic device (100) can determine the extraction order of object C (913), which is placed on the far right, as the third priority. The electronic device (100) can determine the extraction order of object D (921), which is placed with part of it obscured by object C (913), as the fourth priority. The electronic device (100) can determine the extraction order of object E (922), which corresponds to the product information that is the extraction target of the electronic device (100) but is not placed upright, as the fifth priority. The electronic device (100) may not determine the extraction order for object F (931) that is not included in the plurality of objects.

[0145] According to one embodiment, the electronic device (100) can determine the extraction posture of the robot arm for the objects based on the shape and arrangement of the objects. For example, referring to identification number 910, the electronic device (100) can determine the extraction posture of the robot arm so that it can grasp the objects upright for objects A (911), B (912), and C (913) that are arranged upright. The electronic device (100) can determine the extraction posture of the robot arm so that it can grasp object D (921) upright when object C (913) is extracted and only object D (921) remains for object D (921) which is arranged with a portion obscured by object C (913). The electronic device (100) can determine the extraction posture of the robot arm such that the angle of the robot arm is rotated by the angle at which object E (922) is rotated to the right as the extraction posture of the robot arm.

[0146] According to one embodiment, the electronic device (100) can initiate extraction of a plurality of objects by controlling the robot arm based on the extraction order and the extraction posture of the robot arm. For example, in identification number 910, the electronic device (100) can extract object A (911) in the first priority according to the extraction posture of object A (911). The electronic device (100) can extract object B (912) in the second priority according to the extraction posture of object B (912). The electronic device (100) can extract object C (913) in the third priority according to the extraction posture of object C (913).

[0147] According to one embodiment, the electronic device (100) can determine that, while extraction of a plurality of objects is being performed, a designated condition for photographing at least one remaining object among the plurality of objects is satisfied. For example, referring to identification number 910, the electronic device (100) can identify that object D (921), which has an extraction priority of 4th rank, is not extracted in the extraction posture of the robot arm determined for object D (921). This may be because object D (921) was obscured by object C (913) in the first image, so the extraction posture is inaccurate and extraction may not have occurred. Accordingly, the electronic device (100) can determine that a designated condition for photographing object D (921) is satisfied.

[0148] According to one embodiment, the electronic device (100) may obtain a second image by photographing at least one remaining object when it is determined that a specified condition is satisfied. For example, referring to identification number 920, the electronic device (100) may obtain a second image by photographing at least one remaining object, object D (921) and object E (922). Even in this case, the electronic device (100) may select object D (921) and object E (922), which correspond to product information, as a plurality of objects, excluding other object F (931) included in the photographed second image.

[0149] According to one embodiment, the electronic device (100) can extract at least one remaining object based on a second image. The electronic device (100) can determine the extraction order and the extraction posture of the robot arm for at least one remaining object based on the second image, and can initiate extraction for at least one remaining object. For example, referring to identification number 920, the electronic device (100) can determine the extraction order of object D (921) as first priority and the extraction order of object E (922) as second priority based on the second image. The electronic device (100) can determine the extraction posture of the robot arm so as to be able to grasp the object upright with respect to object D (921) which is positioned upright based on the second image, and can determine the extraction posture of the robot arm such that the angle of the robot arm is rotated by the angle of positioned to the right with respect to object E (922). The electronic device (100) can initiate extraction with respect to object D (921) and object E (922).

[0150] According to one embodiment, if the electronic device (100) determines that the specified conditions for shooting are not satisfied, it may extract at least one remaining object based on the second image. For example, referring to identification number 920, the electronic device (100) may successfully extract object D (921) and object E (922) in the process of extracting object D (921) and object E (922) according to the extraction order and extraction posture. At this time, the electronic device (100) may determine that the specified conditions for shooting are not satisfied and may extract object D (921) and object E (922) based on the extraction order determined based on the second image and the extraction posture of the robot arm.

[0151] According to one embodiment, the electronic device (100) may terminate its operation when it has extracted a plurality of objects. For example, referring to identification number 930, the electronic device (100) may terminate its operation because it has extracted all of the plurality of objects. The electronic device (100) may terminate its operation without extracting other objects F (931) that are not objects to be extracted.

[0152] However, examples of an electronic device extracting multiple objects from a mobile loading device loaded with multiple objects are not limited to the examples described above, and the electronic device may extract objects according to a different extraction sequence and a different extraction posture of a robot arm.

[0153] FIG. 10 is a block diagram of an electronic device (100) according to various embodiments of the present disclosure.

[0154] Referring to FIG. 10, according to one embodiment, an electronic device (100) may include a robot arm (104). The robot arm (104) may grasp and extract a plurality of objects. The robot arm (104) may grasp and extract a plurality of objects according to the extraction posture of the robot arm (104) determined by a processor. The robot arm (104) may grasp and extract a plurality of objects sequentially according to the extraction order determined by a processor.

[0155] Referring to FIG. 10, according to one embodiment, a robot arm (104) may include a sensor (105). The sensor (105) may be placed on at least a part of the robot arm (104). The robot arm (104) may collect sensing data during the extraction process of a plurality of objects using the sensor, and a processor may determine whether the extraction is being performed normally. The robot arm (104) may collect sensing information related to whether a plurality of objects have been grasped using the sensor. The robot arm (104) may transmit the collected sensing information to a memory and / or a processor.

[0156] Referring to FIG. 10, according to one embodiment, the sensor (105) may include a position sensor or a pressure sensor. The position sensor can sense information regarding the spatial position and degree of rotation of the robot arm. The pressure sensor can sense information regarding the pressure with which the robot arm is holding an object.

[0157] Referring to FIG. 10, according to one embodiment, an electronic device (100) may include a depth camera (101). The depth camera (101) may capture a plurality of objects to acquire an image. The depth camera may acquire a 3D image containing spatial information. The depth camera (101) may transmit the acquired image to a memory and / or processor.

[0158] Referring to FIG. 10, according to one embodiment, an electronic device (100) may include a memory (103). The memory (103) may store sensing information, images, instructions, and / or product information. The memory (103) may receive and store product information related to the shapes of objects from a user or an external electronic device.

[0159] Referring to FIG. 10, according to one embodiment, the electronic device (100) may include a processor (102).

[0160] Referring to FIG. 10, in one embodiment, the processor (102) may include a shape and arrangement identification unit, an extraction order determination unit, an extraction posture determination unit, and an extraction motion control unit. The shape and arrangement identification unit may identify the shape and arrangement of objects using an image captured by a depth camera. The extraction order determination unit may determine the extraction order of objects based on the shape and arrangement of objects identified by the shape and arrangement identification unit. The extraction posture determination unit may determine the extraction posture of a robot arm based on the extraction order determined by the extraction order determination unit. The extraction motion control unit may initiate an extraction operation using a robot arm. The extraction motion control unit may determine whether a designated condition for re-photography is satisfied during extraction, and if the condition is not satisfied, proceed with the extraction operation sequentially. If the designated condition for re-photography is not satisfied during extraction, the extraction motion control unit may request a re-photography with a depth camera. The extraction operation control unit can terminate the operation or perform a new operation when extraction of multiple objects is completed.

[0161] FIG. 11 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0162] Referring to FIG. 11, in a network environment (1100), an electronic device (1101) may communicate with an electronic device (1102) through a first network (1198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (1104) or a server (1108) through a second network (1199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1101) may communicate with the electronic device (1104) through a server (1108). According to one embodiment, the electronic device (1101) may include a processor (1120), memory (1130), input module (1150), sound output module (1155), display module (1160), audio module (1170), sensor module (1176), interface (1177), connection terminal (1178), haptic module (1179), camera module (1180), power management module (1188), battery (1189), communication module (1190), subscriber identification module (1196), or antenna module (1197). In some embodiments, at least one of these components (e.g., connection terminal (1178)) may be omitted from the electronic device (1101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (1176), camera module (1180), or antenna module (1197)) may be integrated into a single component (e.g., display module (1160)). The electronic device (1101) may correspond to the electronic device (100) of FIG. 10.

[0163] The processor (1120) can, for example, execute software (e.g., program (1140)) to control at least one other component (e.g., hardware or software component) of the electronic device (1101) connected to the processor (1120) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (1120) can store commands or data received from other components (e.g., sensor module (1176) or communication module (1190)) in volatile memory (1132), process the commands or data stored in volatile memory (1132), and store the resulting data in non-volatile memory (1134). According to one embodiment, the processor (1120) may include a main processor (1121) (e.g., a central processing unit or an application processor) or an auxiliary processor (1123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (1101) includes a main processor (1121) and an auxiliary processor (1123), the auxiliary processor (1123) may be configured to use less power than the main processor (1121) or to be specialized for a specified function. The auxiliary processor (1123) may be implemented separately from the main processor (1121) or as part thereof. The processor (1120) may correspond to the processor (102) of FIG. 10.

[0164] The auxiliary processor (1123) may control at least some of the functions or states associated with at least one component of the electronic device (1101) (e.g., display module (1160), sensor module (1176), or communication module (1190)) on behalf of the main processor (1121) while the main processor (1121) is in an inactive (e.g., sleep) state, or together with the main processor (1121) while the main processor (1121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (1123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (1180) or communication module (1190)). According to one embodiment, the auxiliary processor (1123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (1101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (1108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0165] The memory (1130) can store various data used by at least one component of the electronic device (1101) (e.g., processor (1120) or sensor module (1176)). The data may include, for example, input data or output data for software (e.g., program (1140)) and related commands. The memory (1130) may include volatile memory (1132) or non-volatile memory (1134). The memory (1130) may correspond to the memory (103) of FIG. 10.

[0166] The program (1140) may be stored as software in memory (1130) and may include, for example, an operating system (1142), middleware (1144), or an application (1146).

[0167] The input module (1150) can receive commands or data to be used for a component of the electronic device (1101) (e.g., processor (1120)) from outside the electronic device (1101) (e.g., user). The input module (1150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0168] The sound output module (1155) can output a sound signal to the outside of the electronic device (1101). The sound output module (1155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0169] The display module (1160) can visually provide information to an external (e.g., user) of the electronic device (1101). The display module (1160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (1160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0170] The audio module (1170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (1170) can acquire sound through the input module (1150) or output sound through the sound output module (1155) or an external electronic device (e.g., electronic device (1102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (1101).

[0171] The sensor module (1176) can detect the operating state of the electronic device (1101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (1176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor. The sensor module (1176) may correspond to the sensor (105) of FIG. 10.

[0172] The interface (1177) may support one or more specified protocols that can be used for the electronic device (1101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (1102)). According to one embodiment, the interface (1177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0173] The connection terminal (1178) may include a connector through which the electronic device (1101) can be physically connected to an external electronic device (e.g., electronic device (1102)). According to one embodiment, the connection terminal (1178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0174] The haptic module (1179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (1179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0175] The camera module (1180) can capture still images and video. According to one embodiment, the camera module (1180) may include one or more lenses, image sensors, image signal processors, or flashes. The camera module (1180) may correspond to the depth camera (101) of FIG. 10.

[0176] The power management module (1188) can manage the power supplied to the electronic device (1101). According to one embodiment, the power management module (1188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0177] The battery (1189) can supply power to at least one component of the electronic device (1101). According to one embodiment, the battery (1189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0178] The communication module (1190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (1101) and an external electronic device (e.g., electronic device (1102), electronic device (1104), or server (1108)), and the performance of communication through the established communication channel. The communication module (1190) may include one or more communication processors that operate independently of the processor (1120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1190) may include a wireless communication module (1192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (1104) via a first network (1198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (1199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1192) can identify or authenticate the electronic device (1101) within a communication network such as the first network (1198) or the second network (1199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (1196).

[0179] The wireless communication module (1192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (1192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (1192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (1192) can support various requirements specified in the electronic device (1101), external electronic device (e.g., electronic device (1104)), or network system (e.g., second network (1199)). According to one embodiment, the wireless communication module (1192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0180] An antenna module (1197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (1197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (1197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (1198) or a second network (1199), may be selected from the plurality of antennas, for example, by a communication module (1190). A signal or power may be transmitted or received between the communication module (1190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (1197).

[0181] According to various embodiments, the antenna module (1197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0182] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0183] According to one embodiment, commands or data may be transmitted or received between the electronic device (1101) and an external electronic device (1104) through a server (1108) connected to a second network (1199). Each of the external electronic devices (1102, or 1104) may be the same or a different type of device as the electronic device (1101). According to one embodiment, all or part of the operations performed on the electronic device (1101) may be performed on one or more of the external electronic devices (1102, 1104, or 1108). For example, if the electronic device (1101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (1101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (1101). The electronic device (1101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (1101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1104) may include an Internet of Things (IoT) device. The server (1108) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (1104) or server (1108) may be included within the second network (1199). The electronic device (1101) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0184] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0185] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0186] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0187] Various embodiments of the present document may be implemented as software (e.g., program (1140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (1136) or external memory (1138)) readable by a machine (e.g., electronic device (1101)). For example, a processor (e.g., processor (1120)) of the machine (e.g., electronic device (1101)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0188] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0189] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0190] The electronic device according to various embodiments of the present disclosure can achieve the effect of enabling the operation of an efficient object extraction system by extracting all of a plurality of objects using a single image.

[0191] The electronic device according to various embodiments of the present disclosure can achieve the effect of improving productivity by enabling the material to be photographed with a minimum number of shots and the material to be extracted based on the extraction sequence and the extraction posture of the robot arm, even if re-photography is required during the extraction of an object.

[0192] The electronic device according to various embodiments of the present disclosure can be easily applied to large objects to efficiently extract large objects (large materials), thereby allowing for the expectation of reduced labor costs and improved productivity, and can achieve the effect of minimizing work that causes musculoskeletal disorders during the extraction process of large objects.

[0193] The electronic device according to various embodiments of the present disclosure is capable of operating when there is shape data for a plurality of objects through a depth camera, and thus can be easily applied when extracting various objects, so it can be widely utilized in various industrial sites and has versatility.

[0194] An electronic device according to one embodiment of the present disclosure may include a camera, a robot arm, a memory for storing instructions, and at least one processor. At least one processor may acquire a first image by photographing a plurality of objects using the camera when the instructions are executed. At least one processor may identify the shape and arrangement of the plurality of objects using the first image when the instructions are executed. At least one processor may determine the extraction order of the plurality of objects based on the shape and arrangement of the plurality of objects when the instructions are executed. At least one processor may determine the extraction posture of the robot arm for the objects based on the shape and arrangement of the plurality of objects when the instructions are executed. At least one processor may initiate extraction of the plurality of objects by controlling the robot arm based on the extraction order and the extraction posture of the robot arm when the instructions are executed. At least one processor may, when the above instructions are executed, determine whether a specified condition for photographing at least one remaining object among the plurality of objects is satisfied by using a sensor placed on the robot arm while the extraction of the plurality of objects is being performed. If at least one processor determines that the specified condition is not satisfied when the above instructions are executed, it may extract the at least one remaining object based on the first image without photographing the at least one remaining object. If at least one processor determines that the specified condition is satisfied when the above instructions are executed, it may acquire a second image by photographing the at least one remaining object and extract the at least one remaining object based on the second image.

[0195] In one embodiment, the camera may include a depth camera.

[0196] In one embodiment, the plurality of objects may be selected from among a plurality of objects loaded on a mobile loading device. At least one processor may extract product information regarding the shape of the plurality of objects from the memory when the instructions are executed. At least one processor may acquire the first image by photographing the plurality of objects including the plurality of objects when the instructions are executed. At least one processor may select the plurality of objects from among the plurality of objects loaded on the mobile loading device using the product information and the first image when the instructions are executed.

[0197] In one embodiment, at least one processor may determine a score for the plurality of objects based on the shape and arrangement of the plurality of objects when the instructions are executed. At least one processor may determine the extraction order of the plurality of objects based on the score when the instructions are executed.

[0198] In one embodiment, at least one processor can determine the extraction posture of the robot arm by analyzing the first image using an algorithm based on at least one of ICP (Iterative Closest Point), REGTR (End-to-end Point Cloud Correspondences with Transformers), or FPFH (Fast Point Feature Histogram) when the instructions are executed.

[0199] In one embodiment, at least one processor can identify whether the robot arm has grasped the at least one remaining object according to the extraction posture of the robot arm through the sensor placed on the robot arm when the instructions are executed. At least one processor can determine that the specified condition is satisfied if, when the instructions are executed, the at least one remaining object is not grasped by the robot arm according to the extraction posture of the robot arm determined based on the first image.

[0200] In one embodiment, the sensor disposed on the robot arm may include at least one of a pressure sensor or a position sensor. At least one processor can identify whether the robot arm has grasped the at least one remaining object through at least one of the pressure sensor or the position sensor when the instructions are executed.

[0201] In one embodiment, at least one processor can determine whether the specified condition is satisfied after the robot arm repeats the operation of grasping the at least one remaining object a preset number of times when the instructions are executed.

[0202] In one embodiment, at least one processor can determine whether the specified condition is satisfied by comparing the shape of the at least one remaining object included in the first image with the shape of the at least one remaining object obtained from the product information when the instructions are executed.

[0203] In one embodiment, at least one processor can identify the position of the plurality of objects with respect to the X, Y, and Z axes and the degree of rotation of the plurality of objects with respect to the X, Y, and Z axes using the first image when the instructions are executed. At least one processor can identify the arrangement of the plurality of objects using the position and the degree of rotation of the plurality of objects when the instructions are executed.

[0204] A method of operation of an electronic device according to one embodiment of the present disclosure comprises: acquiring a first image by photographing a plurality of objects using a camera; identifying the shape and arrangement of the plurality of objects using the first image; determining the extraction order of the plurality of objects based on the shape and arrangement of the plurality of objects; determining the extraction posture of a robot arm for the objects based on the shape and arrangement of the plurality of objects; initiating extraction of the plurality of objects by controlling the robot arm based on the extraction order and the extraction posture of the robot arm; determining whether a specified condition for photographing at least one remaining object among the plurality of objects is satisfied using a sensor placed on the robot arm while the extraction of the plurality of objects is being performed; and if it is determined that the specified condition is not satisfied, extracting the at least one remaining object based on the first image without photographing the at least one remaining object. And if it is determined that the above specified condition is satisfied, the method may include acquiring a second image by photographing the at least one remaining object and extracting the at least one remaining object based on the second image.

[0205] In one embodiment, in a method of operating an electronic device, the camera may include a depth camera.

[0206] In one embodiment, in a method of operating an electronic device, the plurality of objects may be selected from among a plurality of objects loaded on a movable loading device. In one embodiment, the method of operating an electronic device may include: an operation of extracting product information regarding the shape of the plurality of objects from a memory; an operation of acquiring the first image by photographing the plurality of objects including the plurality of objects; and an operation of selecting the plurality of objects from among the plurality of objects loaded on the movable loading device using the product information and the first image.

[0207] In one embodiment, a method of operating an electronic device may include: determining a score for the plurality of objects based on the shape and arrangement of the plurality of objects; and determining the extraction order of the plurality of objects based on the score.

[0208] In one embodiment, the method of operation of the electronic device may include determining the extraction posture of the robot arm by analyzing the first image using an algorithm based on at least one of ICP (Iterative Closest Point), REGTR (End-to-end Point Cloud Correspondences with Transformers), or FPFH (Fast Point Feature Histogram).

[0209] In one embodiment, the method of operation of an electronic device may include: identifying whether the robot arm has grasped the at least one remaining object according to the extraction posture of the robot arm through the sensor disposed on the robot arm; and determining that the specified condition is satisfied if the at least one remaining object is not grasped by the robot arm according to the extraction posture of the robot arm determined based on the first image.

[0210] In one embodiment, in a method of operation of an electronic device, the sensor disposed on the robot arm may include at least one of a pressure sensor or a position sensor. In one embodiment, the method of operation of the electronic device may include an operation of identifying whether the robot arm has grasped the at least one remaining object through at least one of the pressure sensor or the position sensor.

[0211] In one embodiment, the method of operation of the electronic device may include the operation of determining whether the specified condition is satisfied after the robot arm repeats the operation of grasping the at least one remaining object a preset number of times.

[0212] In one embodiment, the method of operating an electronic device may include an operation of determining whether the specified condition is satisfied by comparing the shape of the at least one remaining object included in the first image with the shape of the at least one remaining object obtained from the product information.

[0213] In one embodiment, a method of operating an electronic device may include: an operation of identifying the position of the plurality of objects with respect to the X, Y, and Z axes and the degree of rotation of the plurality of objects with respect to the X, Y, and Z axes using the first image; and an operation of identifying the arrangement of the plurality of objects using the position and the degree of rotation of the plurality of objects.

[0214] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0215] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the claims or embodiments described in the specification of this disclosure.

[0216] In the present disclosure, the function or operation performed by an electronic device may be performed by one or more processors executing one or more instructions stored in memory. The function or operation of the electronic device mentioned in the present disclosure may be performed by a single processor executing one or more instructions, or by a combination of multiple processors executing one or more instructions. A processor mentioned in the present disclosure is understood to include a circuit for performing operations or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a micro-processor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on chip (SoC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operation of the electronic device described above.

[0217] In the present disclosure, a program (software module, software) may be stored in a random access memory, a non-volatile memory including flash memory, a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic disc storage device, a compact disc-ROM (CD-ROM), digital versatile discs (DVDs), or other forms of optical storage devices, or a magnetic cassette. Alternatively, it may be stored in a memory composed of some or all of these. The memory may be composed of a single storage medium or a combination of multiple storage media. The one or more instructions may be stored in a single storage medium or distributed across multiple storage media.

[0218] Additionally, the above program may be stored on an attachable storage device that can be accessed via a communication network such as the Internet, Intranet, LAN (local area network), WLAN (wide LAN), or SAN (storage area network), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.

[0219] In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression is selected to suit the situation presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural form, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.

[0220] Additionally, in the present disclosure, terms such as “part,” “module,” etc. may be a hardware component, such as a processor or circuit, and / or a software component executed by a hardware component, such as a processor.

[0221] "Parts" and "modules" may be implemented by a program that is stored on an addressable storage medium and can be executed by a processor. For example, "parts" and "modules" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as by processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0222] The specific embodiments described in this disclosure are merely examples and do not limit the scope of this disclosure in any way. For the sake of brevity, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted.

[0223] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, comprising only b, comprising only c, or comprising a combination of two or more (comprising a and b, comprising b and c, comprising a and c, or comprising all of a, b, and c).”

[0224] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.

Claims

1. In an electronic device, camera; Robot arm; Memory for storing instructions; and It includes at least one processor, The above instructions are executed by the above at least one processor, and the electronic device: A first image is obtained by photographing a plurality of objects using the above camera, and Identify the shape and arrangement of the plurality of objects using the first image above, and Based on the shape and arrangement of the plurality of objects, the extraction order of the plurality of objects is determined, and Based on the shape and arrangement of the plurality of objects, the extraction posture of the robot arm for the objects is determined, and Initiating extraction of the plurality of objects by controlling the robot arm based on the extraction sequence and the extraction posture of the robot arm, and While the extraction of the plurality of objects is being performed, a sensor placed on the robot arm is used to determine whether a designated condition for photographing at least one remaining object among the plurality of objects is satisfied, and If it is determined that the above specified conditions are not satisfied, the at least one remaining object is not photographed, and the at least one remaining object is extracted based on the first image. An electronic device that, when it is determined that the above specified condition is satisfied, obtains a second image by photographing the at least one remaining object and extracts the at least one remaining object based on the second image.

2. In Paragraph 1, The above camera is an electronic device including a depth camera.

3. In Paragraph 1, The above plurality of objects are selected from among a plurality of objects loaded on a mobile loading device, and The above instructions are executed by the above at least one processor, and the electronic device: Product information regarding the shapes of the plurality of objects is extracted from the memory, and The first image is obtained by photographing the plurality of objects including the plurality of objects mentioned above, and An electronic device that selects the plurality of objects among the plurality of objects loaded on the movable loading device using the above product information and the above first image.

4. In Paragraph 1, The above instructions are executed by the above at least one processor, and the electronic device: A score for the plurality of objects is determined based on the shape and arrangement of the plurality of objects, and An electronic device that determines the extraction order of the plurality of objects based on the above score.

5. In Paragraph 1, The above instructions are executed by the above at least one processor, and the electronic device: An electronic device for determining the extraction posture of the robot arm by analyzing the first image using an algorithm based on at least one of ICP (Iterative Closest Point), REGTR (End-to-end Point Cloud Correspondences with Transformers), or FPFH (Fast Point Feature Histogram).

6. In Paragraph 1, The above instructions are executed by the above at least one processor, and the electronic device: Identifying whether the robot arm has grasped the at least one remaining object according to the extraction posture of the robot arm through the sensor placed on the robot arm, and An electronic device that determines that the specified condition is satisfied when the at least one remaining object is not caught by the robot arm according to the extraction posture of the robot arm determined based on the first image.

7. In Paragraph 6, The sensor placed on the robot arm includes at least one of a pressure sensor or a position sensor, and An electronic device that identifies whether the robot arm has grasped the at least one remaining object through at least one of the pressure sensor or the position sensor.

8. In Paragraph 7, An electronic device that determines whether the specified condition is satisfied after the robot arm repeats the action of grasping the at least one remaining object a preset number of times.

9. In Paragraph 3, The above instructions are executed by the above at least one processor, and the electronic device: An electronic device that determines whether the specified condition is satisfied by comparing the shape of the at least one remaining object included in the first image with the shape of the at least one remaining object obtained from the product information.

10. In Paragraph 1, Using the first image above, the positions of the plurality of objects based on the X, Y, and Z axes and the degree of rotation of the plurality of objects based on the X, Y, and Z axes are identified, and An electronic device that identifies the arrangement of the plurality of objects by utilizing the position and degree of rotation of the plurality of objects.

11. In a method of operating an electronic device, The operation of acquiring a first image by photographing multiple objects using a camera; An operation to identify the shape and arrangement of the plurality of objects using the first image; An operation to determine the extraction order of the plurality of objects based on the shape and arrangement of the plurality of objects; An operation to determine the extraction posture of a robot arm for the objects based on the shape and arrangement of the plurality of objects; An operation to initiate extraction of the plurality of objects by controlling the robot arm based on the extraction sequence and the extraction posture of the robot arm; An operation to determine whether a specified condition for photographing at least one remaining object among the plurality of objects is satisfied using a sensor disposed on the robot arm while the extraction of the plurality of objects is being performed; If it is determined that the above specified condition is not satisfied, the operation of extracting the at least one remaining object based on the first image without photographing the at least one remaining object; and A method comprising the operation of obtaining a second image by photographing the at least one remaining object when it is determined that the above specified condition is satisfied, and extracting the at least one remaining object based on the second image.

12. In Paragraph 11, The above camera is a method including a depth camera.

13. In Paragraph 11, The above plurality of objects are selected from among a plurality of objects loaded on a mobile loading device, and The operation of extracting product information regarding the shapes of the above plurality of objects from memory; The operation of acquiring the first image by photographing the plurality of objects including the plurality of objects; and A method comprising the operation of selecting a plurality of objects among a plurality of objects loaded on a movable loading device using the above product information and the above first image.

14. In Paragraph 11, An operation to determine a score for the plurality of objects based on the shape and arrangement of the plurality of objects; and A method comprising the operation of determining the extraction order of the plurality of objects based on the above score.

15. In Paragraph 11, The sensor placed on the robot arm includes at least one of a pressure sensor or a position sensor, and An operation to identify whether the robot arm has grasped the at least one remaining object according to the extraction posture of the robot arm through at least one of the pressure sensor or the position sensor; and The method includes an operation of determining that the specified condition is satisfied when the at least one remaining object is not grasped by the robot arm according to the extraction posture of the robot arm determined based on the first image, and A method comprising the operation of the robot arm grasping the at least one remaining object a predetermined number of times and then determining whether the specified condition is satisfied.

Images