Processing device, processing system, processing method, and recording medium

Abstract

This processing device comprises: a planning means for generating a plan to move, to a first region which is in a finite-sized space and in which objects are aligned and stacked, objects from a second region which is in the space and is different from the first region; and a prediction means for moving objects from the second region to the first region on the basis of the plan generated by the planning means and predicting the number of objects in the space.

Description

Processing Apparatus, Processing System, Processing Method, and Recording Medium

[0001] The present disclosure relates to a processing apparatus, a processing system, a processing method, and a recording medium.

[0002] Robots are used in various fields such as automobile factories and logistics warehouses. In many cases, it is desired to automate the operation of robots. In particular, in the case of work handling articles such as inventory work in warehouses, it is necessary to predict the number of articles stacked and stored inside a container. Patent Document 1 discloses a technology related to a learning device that reduces the burden on a store regarding inventory as a related technology.

[0003] Japanese Unexamined Patent Application Publication No. 2021-051494

[0004] By the way, in inventory work and the like related to Patent Document 1, there is a demand for a technology capable of predicting the number of articles stacked and stored inside a container.

[0005] One object of each aspect of the present disclosure is to provide a processing apparatus, a processing system, a processing method, a recording medium, and the like that can solve the above problem.

[0006] According to one aspect of the present disclosure, the processing apparatus includes: planning means for generating a plan to move an object in a second region in a finite-sized space to a first region, the second region being different from the first region which is a region where objects are aligned and stacked; and prediction means for moving the object in the second region to the first region based on the plan generated by the planning means, and predicting the number of objects in the space.

[0007] According to another aspect of the present disclosure, a processing system includes the above-described processing apparatus, and a robot that operates based on a plan generated by the processing apparatus.

[0008] According to another aspect of the present disclosure, the processing method includes generating a plan to move objects from a second region in a space, which is different from a first region in a finite-sized space where objects are aligned and stacked, to the first region, and moving the objects from the second region to the first region based on the generated plan, and predicting the number of objects in the space.

[0009] According to another aspect of the present disclosure, the recording medium stores a program that causes a computer to perform the following actions: generate a plan to move objects in a second region of a space, which is different from a first region in a finite-sized space in which objects are aligned and stacked, to the first region; and, based on the generated plan, move the objects in the second region to the first region and predict the number of objects in the space.

[0010] According to each aspect of this disclosure, it is possible to predict the number of articles stacked and stored inside a container.

[0011] This figure shows an example of the configuration of a processing system according to some embodiments of the present disclosure. This figure shows an example of the configuration of an processing device according to some embodiments of the present disclosure. This figure shows an example of the configuration of a control device according to some embodiments of the present disclosure. This figure shows an example of the configuration of a recognition device according to some embodiments of the present disclosure. This figure shows an example of the configuration of a planning device according to some embodiments of the present disclosure. This figure shows an example of the configuration of a storage device according to some embodiments of the present disclosure. This figure shows an example of the processing flow of a processing system according to some embodiments of the present disclosure. This figure shows an example of a two-dimensional area photographed from directly above inside a tray according to some embodiments of the present disclosure. This figure shows an example of the configuration of an processing device according to some embodiments of the present disclosure. This figure shows an example of the processing flow of an processing device according to some embodiments of the present disclosure. This is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

[0012] The embodiments will be described in detail below with reference to the drawings.

[0013] <Embodiment> The processing system 1 according to the embodiment of the present disclosure is a system that can predict the number of items stacked and stored inside a container.

[0014] (Configuration of the Processing System) Figure 1 is a diagram showing an example of the configuration of a processing system 1 according to some embodiments of the present disclosure. As shown in Figure 1, the processing system 1 comprises a robot 10, a camera 20, a processing device 30, a tray 40, and articles 50a1, 50a2, 50a3, 50a4, and 50a5. Articles 50a1, 50a2, 50a3, 50a4, and 50a5 are sometimes collectively referred to as article 50.

[0015] The robot 10 moves the article 50 under the control of the processing unit 30. The robot 10 is, for example, an arm robot. The robot 10 is equipped with an end effector 101, as shown in Figure 1.

[0016] Camera 20 photographs the items 50 in the tray 40. Camera 20 is installed, for example, on the end effector 101 of the robot 10.

[0017] (Configuration of the Processing Unit 30) Figure 2 is a diagram showing an example of the configuration of a processing unit 30 according to some embodiments of the present disclosure. The processing unit 30 predicts the number (total number) of articles 50 by efficiently moving the articles 50 within a limited space (in this example, inside the tray 40). As shown in Figure 2, the processing unit 30 comprises a control device 301, a recognition device 302, a planning device 303, and a storage device 304.

[0018] Figure 3 shows an example of the configuration of a control device 301 according to some embodiments of the present disclosure. As shown in Figure 3, the control device 301 includes a robot controller 3011.

[0019] The robot controller 3011 generates control signals that realize the robot motion time series calculated by the robot motion planning unit 3032 of the planning device 303 (described later). The robot controller 3011 then controls the robot 10 by outputting the generated control signals to the robot 10.

[0020] Figure 4 shows an example of the configuration of a recognition device 302 according to some embodiments of the present disclosure. As shown in Figure 4, the recognition device 302 includes a concealed area estimation unit 3021, a number prediction unit 3022, and an object position and orientation estimation unit 3023.

[0021] The concealed area estimation unit 3021 defines a three-dimensional coordinate system and predicts the concealed area, which is the region where concealment exists between multiple items 50. The concealed area estimation unit 3021 transmits the concealed area information indicating the predicted concealed area to the storage device 304. Details of the concealed area will be described later.

[0022] The quantity prediction unit 3022 predicts the number of items 50 for each area other than the concealed area (i.e., the non-concealed area). The quantity prediction unit 3022 transmits quantity information indicating the predicted number of items 50 to the storage device 304. Details of the non-concealed area will be described later.

[0023] The object position and orientation estimation unit 3023 estimates the position and orientation of each of the articles 50. The object position and orientation estimation unit 3023 transmits the position and orientation information, which is information indicating the estimated position and orientation of each of the articles 50, to the planning device 303.

[0024] Figure 5 shows an example of the configuration of a planning device 303 according to some embodiments of the present disclosure. As shown in Figure 5, the planning device 303 includes an object movement planning unit 3031 and a robot motion planning unit 3032.

[0025] The object movement planning unit 3031 selects the article 50 to be moved. The object movement planning unit 3031 calculates the position and orientation of the selected article 50 at the destination.

[0026] The robot motion planning unit 3032 calculates the usage action, movement trajectory, and input time series for the robot 10 necessary to realize the movement of the item 50, which will have a position and orientation at the destination calculated by the object movement planning unit 3031. The usage action, movement trajectory, and input time series calculated by the robot motion planning unit 3032 are sometimes collectively referred to as the robot motion time series. The usage action is the type of object movement operation. Examples of object movement operation types include "pushing the object" and "picking and placing the object." If there are two types of object movement operations, "pushing the object" and "picking and placing the object," then the robot motion planning unit 3032 deciding on the usage action is equivalent to the robot motion planning unit 3032 deciding which of the two to execute. The robot motion planning unit 3032 transmits the calculated robot motion time series to the robot controller 3011.

[0027] Figure 6 shows an example of the configuration of a storage device 304 according to some embodiments of the present disclosure. As shown in Figure 6, the storage device 304 includes a region-quantity pair storage unit 3041 and a quantity prediction completion determination unit 3042.

[0028] The area-quantity pair storage unit 3041 stores the predicted number of items for each area other than the concealed area prediction unit 3021 (i.e., the non-concealed area), associating it with the corresponding non-concealed area.

[0029] The quantity prediction completion determination unit 3042 outputs a quantity prediction completion flag when the quantity has been predicted for all areas inside the tray 40.

[0030] The tray 40 has a space in which the articles 50 that are subject to quantity prediction by the processing device 30 are located.

[0031] The item 50 is subject to quantity prediction by the processing device 30. The item 50 is moved by the robot 10.

[0032] The processing performed by the processing system 1 according to the embodiment of this disclosure is merely an example, and the processing performed by the processing system 1 is not limited to the processing described above. For example, the processing system 1 may perform the processing described below.

[0033] (Overview of the processing performed by the processing system) Figure 7 is a diagram showing an example of the processing flow of the processing system 1 according to some embodiments of the present disclosure. Here, first, an overview of the processing performed by the processing system 1 will be described with reference to Figure 7.

[0034] In processing system 1, camera 20 captures the entire finite space inside tray 40 (step S1). For example, under the control of robot controller 3011, end-effector 101 is positioned directly above tray 40, and camera 20 captures the entire finite space inside tray 40.

[0035] The concealed area estimation unit 3021 acquires an image captured by the camera 20. Based on the acquired image, the concealed area estimation unit 3021 predicts a concealed area where concealment exists between multiple items 50, and where one item 50 overlaps with a part of another item 50 (step S2). The concealed area estimation unit 3021 transmits the concealed area information indicating the predicted concealed area to the storage device 304. Details of the concealed area prediction process performed by the concealed area estimation unit 3021 in step S2 will be described later.

[0036] In the finite space inside the tray 40, in areas other than the concealed areas, i.e., the non-concealed areas, the items 50 either do not overlap or are stacked completely on top of each other. Therefore, the quantity prediction unit 3022 can predict the number of items 50 in the non-concealed areas (step S3). In this way, the quantity prediction unit 3022 creates an area-quantity pair by associating the estimated non-concealed areas with the predicted number of items 50 in those areas. Then, the quantity prediction unit 3022 stores the created area-quantity pair in the area-quantity pair storage unit 3041 (step S4). Details of the method for predicting the number of items 50 will be described later.

[0037] Here, the quantity prediction completion determination unit 3042 determines whether it was able to predict the number of all items in all regions (i.e., the total number of all items 50 present in the finite space) (step S5). The quantity prediction completion determination unit 3042 determines that it was able to predict the total number of all items 50 present in the finite space if the region stored by the region-quantity pair storage unit 3041 includes the entire finite space, that is, if the number of items 50 has been predicted for the entire finite space. Otherwise, the quantity prediction completion determination unit 3042 determines that it was not able to predict the total number of all items 50 present in the finite space.

[0038] The quantity prediction completion determination unit 3042 terminates the quantity prediction process if it determines that it has been able to predict the number of items in all areas (YES in step S5).

[0039] Furthermore, if the quantity prediction completion determination unit 3042 determines that it could not predict the number of items in all areas (i.e., the quantity prediction process continues) (NO in step S5), the object position and orientation estimation unit 3023 estimates the position and orientation of the articles 50 based on the images captured by the camera 20 described above (step S6). Here, the object position and orientation estimation unit 3023 does not need to estimate the position and orientation of all articles 50 within the finite space. The object position and orientation estimation unit 3023 estimates the position and orientation only for articles 50 whose surfaces are sufficiently exposed (not concealed).

[0040] The object movement planning unit 3031 performs an object movement plan (step S7). Specifically, the object movement planning unit 3031 selects the item 50 to be moved based on the item position and orientation information estimated by the object position and orientation estimation unit 3023. The object movement planning unit 3031 also calculates the state (position and orientation) of the destination within the finite space based on the item position and orientation information estimated by the object position and orientation estimation unit 3023.

[0041] Here, the robot motion planning unit 3032 executes a motion plan for the robot (step S8). Specifically, in order to realize the movement of the article 50 calculated by the object movement planning unit 3031, the robot motion planning unit 3032 calculates a robot motion time series that realizes the movement of the article 50 described above. Then, the robot motion planning unit 3032 transmits the calculated robot motion time series to the robot controller 3011.

[0042] The robot controller 3011 receives the robot motion time series from the robot motion planning unit 3032. The robot controller 3011 generates a control signal that implements the received robot motion time series. Then, the robot controller 3011 controls (actuates) the robot 10 by outputting the generated control signal to the robot 10 (step S9).

[0043] When the processing of step S9 is completed, the robot controller 3011 returns the processing to step S1. Accordingly, after the motion of the robot 10 is completed, that is, after the movement of the article 50 is completed, the camera 20 captures an image of the inside of the finite space again.

[0044] It should be noted that the number of articles 50 may be either plural or one. Further, the work space targeted by the processing system 1 is not limited to the space described above. For example, the work space targeted by the processing system 1 may be a work space in which the articles 50 are stored inside a shelf. That is, if the articles 50 are stored inside any finite space, that finite space serves as the work space targeted by the processing system 1.

[0045] Furthermore, in the embodiment of the present disclosure described above, the description has been given on the assumption that the camera 20 provided on the hand 101 photographs the tray 40 from directly above in the work space. However, for example, the camera 20 may be a fixed camera installed at a position overlooking the entire work space (a position looking down from the positive direction of the z-axis toward the negative direction of the z-axis). In other words, the camera 20 may be installed and capture images at a position and in a posture that allows visual recognition of the inside of the tray 40 or the shelf.

[0046] (Details of the processing performed by the processing system) Next, we will explain the details of the processing performed by processing system 1. Here, we will explain the details of the processing performed by several processing units in processing system 1.

[0047] The concealed area estimation unit 3021 defines a three-dimensional coordinate system based on the position and orientation of the camera 20 that captures images in a finite space. Specifically, the depth direction of the camera 20 is defined as the Z direction, and the plane perpendicular to the Z direction is defined as the XY plane. The directions of the X and Y axes can be determined arbitrarily. However, in this case, the X and Y axes are determined to be parallel to each side of the tray 40. An example of a three-dimensional coordinate system determined in this way is the coordinate system shown in Figure 1.

[0048] Furthermore, the occluded area estimating unit 3021 divides a two-dimensional area obtained by imaging the inside of the tray 40 from directly above (for example, the area shown in FIG. 8 described later) into an occluded area and a non-occluded area based on an RGB image and a depth image captured by the camera 20. FIG. 8 is a diagram illustrating an example of a two-dimensional area obtained by imaging the inside of the tray 40 from directly above according to some embodiments of the present disclosure. Here, it can be seen that in the area R1 of the plane shown in FIG. 8 (the hatched rectangular area on the plane shown in FIG. 8), the articles 50 partially overlap each other. In this way, among the areas inside the tray 40, an area where the articles 50 overlap each other is referred to as an occluded area. On the other hand, in the area R2 of the plane shown in FIG. 8 (the non-hatched rectangular area on the plane shown in FIG. 8), the articles 50 do not overlap each other. In this way, among the areas inside the tray 40, an area where the articles 50 do not overlap each other (that is, an area that is not an occluded area among the areas inside the tray 40) is referred to as a non-occluded area. In other words, in the non-occluded area, the camera 20 can overlook and recognize the entire article 50 inside the non-occluded area. Alternatively, another article 50 is disposed completely overlapping on the lower side when the direction away from the camera 20 in the thickness direction of the plane shown in FIG. 8 of the recognizable article 50 inside the non-occluded area (that is, the negative direction of the z-axis) is defined as downward. As a method of classifying the occluded area and the non-occluded area as described above, there may be mentioned a method of calculating an object type classification accuracy for each area, and determining the area as an occluded area if the accuracy is equal to or lower than a predetermined threshold. The object type classification accuracy can be calculated by using a method generally called "object detection". For example, object detection functions such as YOLO (You Only Look Once) and YOLOX are publicly available and can be generally used.

[0049] It should be noted that each of the aforementioned occluded area and non-occluded area is assumed to be represented by a two-dimensional rectangle, that is, four real numbers representing the upper and lower limits of coordinates in the X direction and the Y direction respectively. However, each of the occluded area and the non-occluded area may be represented by a set of pixels on an image.

[0050] The quantity prediction unit 3022 predicts the number of items 50 present in each region (a two-dimensional rectangle or a set of pixels) that has been determined to be an unconcealed region by the concealed region estimation unit 3021.

[0051] Specifically, the quantity prediction unit 3022 first identifies the number of items 50 that are fully visible (hereinafter referred to as "unconcealed items 50") using instance segmentation. For example, in the case shown in Figure 8, the quantity prediction unit 3022 identifies the number of unconcealed items 50 photographed by the camera 20 in the unconcealed area R2 shown in Figure 8 as 3. Next, the quantity prediction unit 3022 measures the z-coordinate (i.e., depth) of each unconcealed item 50. Here, the quantity prediction unit 3022 calculates the number N_u, which indicates how many items 50 are hidden below each unconcealed item 50, using equation (1).

[0052]

[0053] Here, z_o is the z-coordinate (depth) of the concealed item 50. Also, z_tray is the z-coordinate (depth) of the pre-set tray 40. And l_z is the thickness of the item 50. In this way, the quantity prediction unit 3022 can predict the number of items 50 that are fully visible in the concealed area R2 and the number of items 50 N_u that are located below them. Therefore, the quantity prediction unit 3022 can predict the total number of items 50 present in the concealed area R2.

[0054] The quantity prediction unit 3022 creates a pair between the predicted quantity and the corresponding hidden area R2 information (a two-dimensional rectangle or a set of pixels). In the following explanation, this pair information may be referred to as the area-quantity pair. Equation (2) shows an example of the area-quantity pair P_i.

[0055]

[0056] Here, i is the ID (ID identification) of the region-quantity pair. For example, if the quantity prediction unit 3022 measures R concealed regions R2, then i = 1, 2, ..., R. x_i^low represents the lower limit of the x-axis direction of the concealed region R2. Also, x_i^upp represents the upper limit of the x-axis direction of the concealed region R2. Similarly, y_i^low represents the lower limit of the y-axis direction of the concealed region R2. Also, y_i^upp represents the upper limit of the y-axis direction of the concealed region R2. z_i represents the z position (i.e., depth) of the surface of the concealed region R2. N_i is the number of objects predicted to exist within the concealed region R2.

[0057] For example, the quantity prediction unit 3022 acquires the depth of each item 50 in the concealed area R2 using the camera 20. The quantity prediction unit 3022 then sets the maximum value of these depths as the corresponding z-position z_i on the surface of the concealed area R2. In this way, the quantity prediction unit 3022 can acquire the z-position z_i on the surface of the concealed area R2.

[0058] The quantity prediction unit 3022 transmits the acquired information P := {P_1, P_2, ..., P_R} of the R region-quantity pairs to the storage device 304.

[0059] The storage device 304 stores the number of hidden areas based on the area-number pair P transmitted from the number prediction unit 3022.

[0060] Here, it should be noted that, as shown in the processing flow in Figure 7, the processing system 1 may repeatedly estimate and predict the number of hidden regions. In the processing system 1, the region-number pair storage unit 3041 stores the region-number pair P transmitted from the number prediction unit 3022 as is during the first number prediction. In subsequent number predictions, if the region-number pair storage unit 3041 contains new regions whose numbers have not yet been stored among the region-number pairs P newly transmitted from the number prediction unit 3022, it stores only the pairs of numbers corresponding to those new regions. Furthermore, when the region-number pair storage unit 3041 stores the number of regions containing the state of the destination of an object planned by the object movement planning unit 3031 (described later), it actually adds 1 to that number.

[0061] The quantity prediction completion determination unit 3042 refers to the hidden empty area stored in the area-quantity pair storage unit 3041. The quantity prediction completion determination unit 3042 outputs a quantity prediction completion flag only if it determines that the referenced hidden empty area encompasses the entire internal area of ​​the tray 40.

[0062] In some embodiments of this disclosure, the object position and orientation estimation unit 3023 actually measures the position and orientation of each article 50 in three-dimensional space (specifically, orientation information in Euler angle representation or quaternion representation). Furthermore, the tray 40, which is an obstacle, does not move in three-dimensional space. Therefore, the object position and orientation estimation unit 3023 does not need to measure the position and orientation of the tray 40. In this case, when designing the system, it is necessary to provide information representing the position, orientation, and shape of the tray 40 as constants to the control device 301, recognition device 302, and planning device 303 of the processing device 30 in advance and store them in the storage device 304 in advance.

[0063] However, it is possible that obstacles such as tray 40 may change each time a robot work plan is performed. In such cases, the object position and orientation estimation unit 3023 needs to measure the position and orientation of the obstacle each time it performs a measurement. The object position and orientation estimation unit 3023 provides the position and orientation information of one or more objects, measured in this manner, to the planning device 303.

[0064] The object movement planning unit 3031 selects an object to be moved based on the concealed area information, the unconcealed area information, and the object position and orientation estimation information provided by the recognition device 302. The object movement planning unit 3031 then calculates the destination position and orientation of the selected object. This process of selecting the object to be moved and calculating its position and orientation is specifically carried out in the following steps.

[0065] First, the object movement planning unit 3031 extracts position and orientation information for all objects belonging to all detected concealed areas. Then, the object movement planning unit 3031 designates the objects corresponding to the extracted position and orientation information as candidate objects to be moved. Let's assume that the number of candidate objects to be moved extracted in this way is M. Then, the object movement planning unit 3031 calculates a total of MN candidate destination states by generating multiple (let's assume N) candidate destination states (position and orientation) for each candidate object to be moved. A method for calculating candidate destination states is random sampling. Note that the method for calculating candidate destination states may be other than random sampling.

[0066] Next, the object movement planning unit 3031 calculates the value of the evaluation function J(x_0, x_1) for each of the calculated candidate states of the destination. The evaluation function J(x_0, x_1) is defined as shown in equation (3).

[0067]

[0068] Here, x_0 represents the position and orientation of the object before movement. x_1 represents the position and orientation of the object after movement. The function F(x_0, x_1) is the magnitude of the step change associated with the movement. This magnitude is the magnitude (absolute value) of the vertical movement (z-direction) between x_0 and x_1. The function G(x_0, x_1) is a function that evaluates the feasibility of the movement of the object. Specifically, the object movement planning unit 3031 can use pre-trained predictions of the success or failure of robot movements as the function G(x_0, x_1). Existing technologies may be used as a method for pre-training such predictions of the success or failure of robot movements.

[0069] Next, the object movement planning unit 3031 selects the one with the smallest J(x_0, x_1) value from among MN candidate destination states. The object movement planning unit 3031 then uses this selection as the final plan result for the target object and its destination state.

[0070] Here, minimizing the function F(x_0, x_1) leads to aiming for safe and efficient movement of objects within a finite space. When moving objects within a finite space such as the inside of tray 40, if objects are stacked or unstacked haphazardly, the arrangement of objects inside tray 40 becomes unstable. As a result, there is a high risk that objects may spill out of tray 40 or that the arrangement of objects inside tray 40 will collapse, damaging the objects. The object movement planning unit 3031 can eliminate such risks in advance by minimizing the function F(x_0, x_1). Furthermore, minimizing the function F(x_0, x_1) also leads to aiming for more efficient (executable in a shorter time) movement of objects. Generally, when moving objects using the end effector of a robot arm, two types of actions performed by the robot are known: pick-and-place actions, which involve grasping and transporting objects using a gripper hand or vacuum hand, and push actions, which involve pushing objects using the end effector of the robot arm. Generally, pick-and-place operations take longer than push operations. Therefore, pick-and-place operations are less efficient than push operations. On the other hand, push operations cannot move objects with varying numbers of steps (objects cannot be moved vertically). The object movement planning unit 3031 minimizes the function F(x_0, x_1) to utilize the efficient push operation as much as possible.

[0071] Minimizing the function G(x_0, x_1) has the effect of ensuring the safety of robot movements. For example, when a robot 10's end effector 101 is used to push an object in the real world, uncertain factors such as friction may prevent the desired object from moving. To minimize this difference between the ideal and reality, it is important for the object movement planning unit 3031 to pre-train a success / failure predictor to predict whether the object will move successfully or not, and to minimize the predicted failure probability.

[0072] Furthermore, if there are no areas without concealment (the entire interior of tray 40 is a concealed area), the object movement planning unit 3031 randomly selects one item 50 from which position and orientation information has been acquired. Then, the object movement planning unit 3031 removes the washed item 50 and places it outside tray 40.

[0073] The object movement planning unit 3031 transmits the calculated state of the object to be moved and its destination to the robot motion planning unit 3032.

[0074] The robot motion planning unit 3032 plans the movement trajectory and actions (such as starting / ending gripping, starting / ending pushing) of the end effector 101 of the robot 10, based on the target movement state of the object transmitted from the object movement planning unit 3031 and the initial state of the object transmitted from the recognition mechanism, in order to realize the movement of the object connecting those two states. Here, the movement trajectory of the end effector 101 of the robot 10 is, for example, a time series of states that the end effector 101 of the robot 10 should take (three-dimensional position and attitude expressed as Euler angles or quaternions). Furthermore, the actions of the end effector 101 of the robot 10 specifically refer to instructions for the end effector 101 of the robot 10 to exert some kind of action on the object, such as gripping (picking), releasing (placing), or pushing. Here, the robot motion planning unit 3032 has been described as calculating the movement trajectory of the end effector 101 of the robot 10. However, the robot motion planning unit 3032 may calculate the time series of the joint angle states of the robot 10, rather than the end effector 101 of the robot 10. Specific methods for performing such planning include robot motion planning methods using optimization algorithms. Examples of optimization algorithms include "integer mixed linear optimization" and "nonlinear continuous optimization."

[0075] The robot motion planning unit 3032 transmits the calculated movement trajectory and actions (such as start / end of suction, start / end of gripping, etc.) of the end-effector 101 of the robot 10 to the robot controller 3011.

[0076] The robot controller 3011 calculates the movement trajectory of the robot's end effector 101 and the input information to the robot 10 body necessary to perform the action, as transmitted from the robot motion planning unit 3032. Then, the robot controller 3011 uses the calculated input information to actually control the movement of the robot 10.

[0077] In this way, processing system 1 can realize the movement of the object described above. After realizing the movement of the object, processing system 1 returns to the process of estimating the concealed region.

[0078] (Advantages) The processing system 1 according to the embodiment of the present disclosure has been described above. In the processing device 30 (an example of a processing device) provided in the processing system 1, the planning device 303 (an example of a planning means) generates a plan to move articles 50 in a concealed area (an example of a second area) in a space that is different from a concealed area (an example of a first area), which is an area in a finite-sized space where articles 50 (an example of an object) are aligned and stacked, to the concealed area. The recognition device 302 (an example of a prediction means) moves the articles 50 in the concealed area to the concealed area and predicts the number of articles 50 in the space based on the plan generated by the planning device 303.

[0079] The processing device 30 included in this processing system 1 can predict the number of items stacked and stored inside the container.

[0080] An aligned and stacked state refers to a state where items overlap on top of each other, for example, where the lower items are stacked in the same orientation as the upper items. In this case, by recognizing the position and orientation of the upper items, it becomes possible to predict the position and number of items hidden underneath. This allows for efficient quantity prediction.

[0081] A processing apparatus 700 according to some embodiments of the present disclosure will be described. Figure 9 is a diagram showing an example of the configuration of a processing apparatus 700 according to some embodiments of the present disclosure. As shown in Figure 9, the processing apparatus 700 according to some embodiments of the present disclosure includes a planning means 701 and a prediction means 702.

[0082] The planning means 701 generates a plan to move objects from a second region in a finite-sized space, which is different from the first region, which is a region in the space where objects are aligned and stacked, to the first region. The prediction means 702 moves the objects from the second region to the first region based on the plan generated by the planning means 701 and predicts the number of objects in the space.

[0083] The processing unit 700 can be implemented, for example, using the functions of the processing unit 30 illustrated in Figures 1 and 2. The planning means 701 can be implemented, for example, using the functions of the planning device 303 illustrated in Figures 2 and 5. The prediction means 702 can be implemented, for example, using the functions of the recognition device 302 illustrated in Figures 2 and 4.

[0084] Next, the processing performed by the processing apparatus 700 according to some embodiments of the present disclosure will be described. Figure 10 is a diagram showing an example of the processing flow of the processing apparatus 700 according to some embodiments of the present disclosure. Here, the processing performed by the processing apparatus 700 will be described with reference to Figure 10.

[0085] In the processing apparatus 700, the planning means 701 generates a plan to move objects from a second region in a space, which is different from a first region in a finite-sized space where objects are aligned and stacked, to the first region (step S101). Based on the plan generated by the planning means 701, the prediction means 702 moves the objects from the second region to the first region and predicts the number of objects in the space (step S102).

[0086] This processing device 700 makes it possible to predict the number of items stacked and stored inside the container.

[0087] In addition, the order of processing in the embodiments of this disclosure may be changed, as long as appropriate processing is performed.

[0088] Although embodiments of this disclosure have been described, the processing system 1, robot 10, camera 20, processing unit 30, and other control devices described above may have a computer device inside. The processing steps described above are stored in the form of a program on a computer-readable recording medium, and the processing is performed when the computer reads and executes this program. A specific example of a computer is shown below.

[0089] Figure 11 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in Figure 11, the computer 5 comprises a CPU (Central Processing Unit) 6, main memory 7, storage 8, and interface 9. For example, the processing system 1, robot 10, camera 20, processing unit 30, and other control devices are each implemented in the computer 5. The operation of each processing unit is stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8, loads it into the main memory 7, and executes the above processing according to the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to each of the storage units described above, according to the program.

[0090] Examples of storage 8 include HDD (Hard Disk Drive), SSD (Solid State Drive), magnetic disk, magneto-optical disk, CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), and semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of the computer 5, or an external medium connected to the computer 5 via interface 9 or a communication line. Furthermore, if this program is distributed to the computer 5 via a communication line, the computer 5 that receives the distribution may expand the program into main memory 7 and execute the above processing. In at least one embodiment, storage 8 is a tangible storage medium that is not temporary.

[0091] Furthermore, the above program may implement some of the functions described above. Moreover, the above program may be a file that can implement the above functions in combination with a program already recorded on the computer device, a so-called differential file (differential program).

[0092] While several embodiments of this disclosure have been described, these embodiments are illustrative and do not limit the scope of the disclosure. These embodiments may be modified in various ways, without departing from the gist of the disclosure.

[0093] Furthermore, some or all of the above embodiments may also be described as follows, but are not limited to these.

[0094] (Note 1) A processing apparatus comprising: planning means for generating a plan to move objects in a second region of a space, which is different from a first region in a finite-sized space in which objects are aligned and stacked, to the first region; and prediction means for moving objects in the second region to the first region and predicting the number of objects in the space based on the plan generated by the planning means.

[0095] (Note 2) The processing apparatus according to Note 1, further comprising: determination means for determining whether or not an object in the second region has finished moving to the first region, wherein the planning means generates a plan for moving an object in the second region, which is a region that is not the first region, to the first region until the determination means determines that an object in the second region has finished moving to the first region.

[0096] (Note 3) The determination means outputs a completion flag indicating the completion of the prediction process for predicting the number of objects when the determination means determines that it has finished moving the objects in the second region to the first region, as described in Note 2.

[0097] (Appendix 4) The processing apparatus according to any one of Appendix 1 to 3, comprising: control means for controlling a robot that moves the object based on the plan generated by the planning means.

[0098] (Note 5) The processing apparatus according to any one of Notes 1 to 4, comprising: a selection means for selecting an object to be moved in such a way as to minimize the number of stacked objects, wherein the planning means generates a plan for moving the object selected by the selection means to the first region.

[0099] (Appendix 6) A processing system comprising: a processing device described in any one of Appendix 1 to 5; and a robot that operates based on a plan generated by the processing device.

[0100] (Note 7) A processing method comprising: generating a plan to move objects in a second region of a space, which is different from a first region in a finite-sized space where objects are aligned and stacked, to the first region; and, based on the generated plan, moving the objects in the second region to the first region and predicting the number of objects in the space.

[0101] (Note 8) The processing method described in Note 7, which includes determining whether or not the object in the second region has finished moving to the first region, and generating a plan to move the object in the second region, which is a region that is not the first region, to the first region until it is determined that the object in the second region has finished moving to the first region.

[0102] (Note 9) The processing method according to Note 8, which includes: generating a plan to move objects in a second region of a space, which is different from a first region in a finite-sized space where objects are aligned and stacked, to the first region; and moving objects in the second region to the first region based on the generated plan and predicting the number of objects in the space.

[0103] (Note 10) A processing method according to any one of Notes 7 to 9, which includes controlling a robot to move the object based on the generated plan.

[0104] (Note 11) A processing method according to any one of Notes 7 to 10, which includes selecting an object to be moved in order to minimize the number of stacked objects, and generating a plan to move the selected object to the first region.

[0105] (Note 12) A recording medium that stores a program that causes a computer to perform the following actions: generate a plan to move objects from a second region in a space, which is different from a first region in a finite-sized space where objects are aligned and stacked, to the first region; and, based on the generated plan, move the objects from the second region to the first region and predict the number of objects in the space.

[0106] (Note 13) A recording medium described in Note 12 that stores a program that causes the computer to perform the following actions: determine whether or not the object in the second region has finished moving to the first region, and generate a plan to move the object in the second region, which is a region that is not the first region, to the first region until it is determined that the object in the second region has finished moving to the first region.

[0107] (Note 14) A recording medium described in Note 13 that stores a program that causes the computer to output a completion flag indicating the completion of a prediction process for predicting the number of objects when it is determined that the objects in the second region have finished moving to the first region.

[0108] (Note 15) A recording medium described in any one of Notes 12 to 14, which stores a program that causes the computer to perform the following actions: control a robot to move the object based on the generated plan.

[0109] (Note 16) A recording medium described in any one of Notes 12 to 15, which contains a program that causes the computer to perform the following actions: selecting an object to be moved in order to minimize the number of stacked objects; and generating a plan to move the selected object to the first region.

[0110] According to each aspect of this disclosure, it is possible to predict the number of articles stacked and stored inside a container.

[0111] 1... Processing system 5... Computer 6... CPU 7... Main memory 8... Storage 9... Interface 10... Robot 20... Camera 30, 700... Processing unit 40... Tray 50, 50a1, 50a2, 50a3, 50a4, 50a5... Item 101... End effector 301... Control device 302... Recognition device 303... Planning device 304... Memory device 3011... Robot controller 3021... Concealed area estimation unit 3022... Quantity prediction unit 3023... Object position and orientation estimation unit 3031... Object movement planning unit 3032... Robot motion planning unit 3041... Area-quantity pair storage unit 3042... Quantity prediction completion determination unit

Claims

1. A processing apparatus comprising: planning means for generating a plan to move objects in a second region of a space, which is different from a first region in a finite-sized space where objects are aligned and stacked, to the first region; and predicting means for moving objects in the second region to the first region and predicting the number of objects in the space based on the plan generated by the planning means.

2. The processing apparatus according to claim 1, comprising: determination means for determining whether or not an object in the second region has finished moving to the first region, wherein the planning means generates a plan for moving an object in the second region, which is a region that is not the first region, to the first region until the determination means determines that an object in the second region has finished moving to the first region.

3. The processing apparatus according to claim 2, wherein the determination means determines that the objects in the second region have finished moving to the first region, and outputs a completion flag indicating the completion of the prediction process for predicting the number of objects.

4. The processing apparatus according to claim 1, further comprising: control means for controlling a robot that moves the object based on the plan generated by the planning means.

5. The processing apparatus according to claim 1, comprising: selection means for selecting an object to be moved in such a way as to minimize the number of stacked objects, wherein the planning means generates a plan for moving the object selected by the selection means to the first region.

6. A processing system comprising: a processing device according to any one of claims 1 to 5; and a robot that operates based on a plan generated by the processing device.

7. A processing method comprising: generating a plan to move objects from a second region in a space, which is different from a first region in a finite-sized space where objects are aligned and stacked, to the first region; and, based on the generated plan, moving the objects from the second region to the first region and predicting the number of objects in the space.

8. A recording medium storing a program that causes a computer to execute: a planning means for generating a plan to move objects in a second region of a space, which is different from a first region in a finite-sized space where objects are aligned and stacked, to the first region; and a prediction means for moving objects in the second region to the first region and predicting the number of objects in the space based on the plan generated by the planning means.

Images