Robot system design device, program, and robot system design method

Abstract

This robot system design device 100 comprises: an object recognition unit 122 for recognizing an object from a video obtained by capturing a state in which a worker repeatedly performs work; an object 3D orientation identification unit 123 for identifying the orientation of the object from the video; a 3D skeleton extraction unit 121 for extracting the skeleton of the worker from the video; an action identification unit 124 for identifying a plurality of actions performed by the worker from the object, the orientation, and the skeleton; a work procedure identification unit 125 for identifying a work procedure according to two or more actions that are repeatedly performed in the plurality of actions; a work content identification unit 126 for identifying work content according to the work procedure, as well as the object, the orientation, and the skeleton at the time of the work procedure; a robot and a peripheral device with which it is possible to realize the work content; and a robot system design unit 130 for generating design information for a robot system, the design information indicating at least the action paths of the robot and the peripheral device.

Description

Robot system design device, program, and robot system design method

[0001] The present disclosure relates to a robot system design device, a program, and a robot system design method.

[0002] Conventionally, the design of robot systems has been carried out by skilled robot SI engineers, but against the backdrop of the rapidly increasing demand for automation, automation of the design work is required.

[0003] For example, Patent Document 1 discloses a design device that enables a user to input information about a workpiece and the processing to be performed on the workpiece according to a screen, and then perform equipment design and cost estimation.

[0004] Japanese Patent Application Laid-Open No. 2023-140545

[0005] However, in the conventional technology, the user has to input all the information necessary for equipment design and cost estimation, which places a very large burden on the user when performing equipment design and cost estimation.

[0006] Therefore, one or more aspects of the present disclosure aim to enable a robot system to be automatically designed from an image capturing the state of an operator performing work.

[0007] A robot system design device according to one aspect of the present disclosure includes an object recognition unit that recognizes an object from an image capturing the state of an operator repeatedly performing work, a posture identification unit that identifies the posture of the object from the image, a skeleton extraction unit that extracts the skeleton of the operator from the image, an action identification unit that identifies a plurality of actions of the operator from the object, the posture, and the skeleton, a work procedure identification unit that identifies a work procedure, which is the procedure of the work, from two or more actions that are repeatedly performed among the plurality of actions, a work content identification unit that identifies a work content, which is the content of the work, from the work procedure and the object, the posture, and the skeleton at the time of the work procedure, a robot and peripheral devices capable of realizing the work content, and a robot system design unit that generates design information of the robot system at least showing an operation path of the robot and the peripheral devices.

[0008] A program according to one aspect of the present disclosure is characterized in that it causes a computer to function as: an object recognition unit that recognizes an object from video footage of an operator repeatedly performing a task; an orientation identification unit that identifies the orientation of the object from the video footage; a skeleton extraction unit that extracts the skeleton of the operator from the video footage; an action identification unit that identifies multiple actions of the operator from the object, the orientation, and the skeleton; an action procedure identification unit that identifies an action procedure, which is the sequence of the task, based on two or more actions that are repeatedly performed in the multiple actions; an action content identification unit that identifies an action content, which is the content of the task, based on the action procedure and the object, the orientation, and the skeleton at the time of the action procedure; and a robot system design unit that generates robot system design information that indicates at least a robot and peripheral devices capable of realizing the action content, and the action paths of the robot and peripheral devices.

[0009] A robot system design method according to one aspect of the present disclosure is characterized by recognizing an object from video footage of an operator repeatedly performing a task, identifying the posture of the object from the video, extracting the operator's skeleton from the video, identifying multiple actions of the operator from the object, posture and skeleton, identifying a work procedure, which is the sequence of the task, based on two or more actions that are repeatedly performed in the multiple actions, identifying the content of the task, which is the content of the task, based on the work procedure and the object, posture and skeleton at the time of the work procedure, and generating robot system design information that shows at least a robot and peripheral devices capable of realizing the work content, and the operating paths of the robot and peripheral devices.

[0010] According to one or more aspects of this disclosure, a robot system can be automatically designed from video footage of an operator performing a task.

[0011] This is a schematic block diagram showing the configuration of the robot system design device according to Embodiment 1. This is a schematic diagram showing examples of various types of information used in the robot system design device. (A) to (E) are schematic diagrams for explaining the posture of an object. This is a schematic block diagram showing the configuration of the PC. This is a flowchart showing the operation of the video acquisition unit and the work information generation unit. This is a flowchart showing the operation of the robot system design unit. This is a schematic block diagram showing the configuration of the robot system design device according to Embodiment 2.

[0012] Embodiment 1. Figure 1 is a schematic block diagram showing the configuration of a robot system design apparatus 100 according to Embodiment 1. The robot system design apparatus 100 comprises an image acquisition unit 110, a work information generation unit 120, and a robot system design unit 130.

[0013] The video acquisition unit 110 acquires video data showing images of an operator repeatedly performing a task. For example, the video acquisition unit 110 may acquire video data from an imaging device such as a camera via a communication unit realized by a communication I / F (Interface) 14, which will be described later, or it may acquire video data from a server connected to a network such as the Internet via the communication unit, or it may acquire video data by reading video data stored in a storage unit (not shown). The acquired video data is provided to the work information generation unit 120.

[0014] Figure 2 is a schematic diagram showing examples of various types of information used in the robot system design device 100. The video data 140 acquired by the video acquisition unit 110 is identified, for example, by video identification information, namely movID. In addition, each frame 141 contained in the video data 140 is identified, for example, by frame identification information, namely frameID.

[0015] The work information generation unit 120 shown in Figure 1 identifies the work content, which is the content of the worker's work, from video data and generates work information that at least indicates that work content. The work information generation unit 120 includes a 3D (three dimensions) skeleton extraction unit 121, an object recognition unit 122, an object 3D posture identification unit 123, an action identification unit 124, a work procedure identification unit 125, and a work content identification unit 126.

[0016] The 3D skeleton extraction unit 121 is a skeleton extraction unit that extracts the skeleton of a worker from the video data shown in the video data. Here, the 3D skeleton extraction unit 121 extracts 3D keypoint information, which indicates the 3D coordinates of keypoints that are feature points corresponding to the skeleton of the worker, such as the body, hands, face, and feet, from each frame contained in the video data, as skeleton information. For example, the 3D skeleton extraction unit 121 may extract 3D keypoint information using a known method such as RTMW (Real-Time Multi-Person 2D and 3D Whole-body Pose Estimation).

[0017] Here, as shown in Figure 2, for each "skeletonID," which is worker identification information that identifies the worker included in the frame 141, a "3DKeypoint" is extracted, which is the 3D coordinate of each of the multiple keypoints. The "3DKeypoint" indicates the 3D coordinate for each of the multiple keypoints that correspond to multiple parts (e.g., body, hands, face, and feet) extracted as the worker's skeleton.

[0018] The object recognition unit 122 shown in Figure 1 recognizes objects from the video data. Here, the object recognition unit 122 detects objects such as parts and tools from each frame included in the video data, recognizes the detected objects, and generates object information representing those objects. For example, the object recognition unit 122 can detect and recognize objects using a known general object recognition model such as YOLOX (You Only Look Once X). If the number of recognizable objects is small, additional training may be performed on the general object recognition model beforehand.

[0019] Here, as shown in Figure 2, for each object contained in the frame 141, object information 143 recognizes object identification information "objID" which identifies the object, "category" which is the type of object, and "segmentation / bbox" which indicates the position of the object's segmentation or bounding box. The type of object is assumed to be either "PartObject" which indicates a part or "ToolObject" which indicates a tool.

[0020] The object 3D pose identification unit 123 shown in Figure 1 is a pose identification unit that identifies the pose of an object from the video data. Here, the object 3D pose identification unit 123 identifies the pose of the object recognized by the object recognition unit 122 for each frame and generates pose information indicating that pose. Here, the pose is identified by the position and orientation of the object. For example, the object 3D pose identification unit 123 can identify the object's pose using known methods such as Epro-PnP (Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation) or HOISDF (Constructing 3D Hand-Object Pose Estimation with Global Signed Distance Fields).

[0021] Figures 3(A) to 3(E) are schematic diagrams illustrating the orientation of an object as determined by the object 3D orientation determination unit 123. As shown in Figure 3(A), the position and orientation of the first object OB1 are determined in a certain frame using the XYZ coordinate system CS. Also, as shown in Figure 3(B), the position and orientation of the second object OB2 are determined in a different frame using the XYZ coordinate system CS.

[0022] Then, the position and orientation of the first object OB1 are changed as shown in the XYZ coordinate system CS in Figure 3(C), and the position and orientation of the second object OB2 are also changed as shown in the XYZ coordinate system CS in Figure 3(D), so that the first object OB1 and the second object OB2 are assembled as shown in Figure 3(E). Thus, the movement paths of the first object OB1 and the second object OB2 during their assembly can be identified.

[0023] Here, as shown in the posture information 144 in Figure 2, the object 3D posture identification unit 123 identifies "timestamp," which is time information corresponding to the frame in which the object was recognized; "position," which indicates the position of the recognized object; "orientation," which indicates the orientation of the recognized object; and "shape," which indicates the shape of the recognized object.

[0024] The motion identification unit 124 shown in Figure 1 identifies multiple actions of the worker from the object indicated by the object information, the posture indicated by the posture information, and the skeleton indicated by the skeleton information. Here, the motion identification unit 124 recognizes the worker's actions frame by frame from the 3D keypoint information as skeleton information, the object information, and the posture information. The actions recognized here are eight basic actions classified as Category 1 of Therblig analysis. These eight basic actions are "extend," "grasp," "carry," "release," "position," "use," "assemble," and "disassemble."

[0025] For example, the motion identification unit 124 can identify basic movements based on rules or learning, using the positions of body parts and objects in combination, or the positions of objects relative to each other. Specifically, there are rules such as a rule that determines "extend" when the hand is gradually moving toward an object, a rule that determines "grasp" when the hand reaches the same position as the object after the "extend" movement toward the object and remains in that position for a predetermined time or longer, and a rule that determines "carry" when the hand and the object move at the same distance for a predetermined time or longer after the "grasp" movement of the object. In this way, rules are predetermined for each basic movement using the relative positions of body parts and objects, or the relative positions of objects relative to each other, and when such rules are met, the motion identification unit 124 can determine that the corresponding basic movement has been performed.

[0026] The motion identification unit 124 determines whether each combination of body parts and objects, as well as each combination of objects, satisfies the above rules. The motion identification unit 124 may also determine whether the above rules are satisfied using only combinations that have predetermined relationships, such as contact relationships, gaze direction relationships, or motion target relationships, in other words, combinations that are determined to have an interaction.

[0027] Whether or not there is interaction can be determined by rules. For example, a rule could be established to determine that contact exists if the position of a body part and the position of an object are within a predetermined range for a predetermined time or longer. A rule could be established to determine that contact exists if the position of an object and the position of another object are within a predetermined range for a predetermined time or longer. A rule could be established to determine that an object is in the direction of the subject's line of sight if its position is within a predetermined range relative to the direction of the face for a predetermined time or longer. A rule could be established to determine that an object is in the direction of the subject's line of sight if its position is within a predetermined range relative to the direction of the subject's line of sight for a predetermined time or longer. A rule could be established to determine that an object is the object of action if its position is within a predetermined range relative to the direction of the subject's arm. A rule could be established to determine that an object is the object of action if its position is within a predetermined range relative to the direction the subject's index finger is pointing.

[0028] Furthermore, whether or not an interaction exists may be determined on a learning basis. For example, if there is a predetermined relationship such as a contact relationship, a gaze direction relationship, or an action target relationship, an interaction may be assumed to exist. In such cases, a learning model may be pre-trained that takes the position of a person's body part and an object, or the positions of objects together, as input data, and outputs whether or not an interaction exists based on such input data. The action identification unit 124 may then use such a learning model to determine whether or not an interaction exists.

[0029] Furthermore, the motion identification unit 124 may use time-series data showing the positions of body parts and objects in a time series, and time-series data showing the positions of objects in a time series, as examples, and use training data in which the actions at the positions of body parts and objects, and the actions at the positions of objects in a time series are correct answers, to input time-series data showing the positions of body parts and objects in a time series, or time-series data showing the positions of objects in a time series, as input data to a pre-trained learning model, thereby acquiring the actions corresponding to the input data. Here, the motion identification unit 124 may identify actions from the learning model using time-series data of the relative positions as input data for each of the above combinations. In this case, the motion identification unit 124 may identify an action as being performed by an operator if the confidence score of the identified action is above a predetermined threshold.

[0030] The operation identification unit 124 may input time series data for all combinations into the learning model, or it may identify the operation by inputting only the time series data for combinations that have interactions as described above into the learning model.

[0031] Then, as shown in Figure 2, the action identification unit 124 generates action information 145 for each frame, which is linked to the ActionID, which is action identification information indicating the recognized action, and identifies the type of action, such as "ReachAction", "GraspAction", "TransportLoadedAction", "ReleaseLoadedAction", "PositionAction", "UseAction", "AssembleAction", or "DisassembleAction".

[0032] As described above, the motion identification unit 124 may identify multiple actions based on the time series of the positions of predetermined parts of the worker included in the skeleton and the positions of objects, or the time series of the positions of objects themselves. Alternatively, if there is a predetermined interaction, the motion identification unit 124 may identify multiple actions based on the time series of the positions of predetermined parts of the worker included in the skeleton and the positions of objects, or the time series of the positions of objects themselves.

[0033] The work procedure identification unit identifies a work procedure, which is the sequence of operations performed by an operator, based on two or more repeated operations in a plurality of operations identified by the operation identification unit 124. First, the work procedure identification unit 125 identifies an operation sequence by arranging the operations recognized by the operation identification unit 124 in chronological order. Then, the work procedure identification unit 125 identifies a work procedure, which is an operation cycle for a single product, from that operation sequence. For example, if the operation sequence is Action = {..., A, B, C, A, B, C, A, B, C, ...}, then one cycle can be inferred to be one of {A, B, C}, {B, C, A}, or {C, A, B}. If the first basic operation is B, then the cycle to be extracted can be uniquely determined as {B, C, A}. The work procedure identification unit 125 identifies the operations of one cycle determined in this way as a single work procedure. Note that the first basic operation may be identified from video data, or it may be predetermined for each operation.

[0034] The work content identification unit 126 identifies the work content, which is the content of the work performed by the worker, based on the work procedure identified by the work procedure identification unit 125 and the object, posture, and skeleton at the time of that work procedure. Here, the work content identification unit 126 identifies the work content by identifying the necessary information from 3D keypoint information, object information, and posture information corresponding to the work procedure identified by the work procedure identification unit 125 and the frames from which the actions included in that work procedure are extracted, and generates work information indicating that work content.

[0035] For example, work information includes information such as the operation, the sequence of operations, the parts used in the operation and the location for accessing those parts, and the gripping state indicating which part of the part is being gripped. The location for accessing the component includes, for example, the location for gripping the part, the location for releasing the part, and the location for attaching the part. In addition, work information may include information that identifies the movement path of the part and the processing performed by the worker's hand or tool. Here, processing is identified by one or more operations, and processing identification rules, which are rules for identifying the processing, are predetermined. Processing identification rules can be defined, for example, by the relative positions of the part and the hand or tool. Furthermore, if the work procedure includes a basic operation of combining multiple parts, the order and orientation of the combination may also be included in the work information.

[0036] As a result, as shown in Figure 2, the work content identification unit 126 can generate work information 146 that identifies the work content for one cycle of work time, cycleTime.

[0037] The robot system design unit 130 shown in Figure 1 designs a robot system from the work information generated by the work information generation unit 120. Here, the robot system design unit 130 generates design information that shows at least a robot and peripheral devices capable of realizing the work content indicated in the work information, and the operating paths of the robot and its peripheral devices. The robot system design unit 130 includes a robot database (hereinafter referred to as robot DB) 131, a peripheral device database (hereinafter referred to as peripheral device DB) 132, a robot selection unit 133, a peripheral device selection unit 134, a layout generation unit 135, a production simulator 136, and a report generation unit 137.

[0038] Robot DB131 is a robot information storage unit that stores robot information indicating the performance and price of each robot for multiple candidate robots. Robot performance information includes, for example, executable movements, operating range, and degrees of freedom of movement, which is necessary when selecting one or more robots from work information.

[0039] Specifically, robot information includes kinematics information that shows the connection relationships between links and the rotation axes of the links; dynamics information that identifies the weight of each link and the speed at which the links operate; and shape information for each link. Furthermore, robot performance may include the robot's external dimensions. In addition, robot information may also include information about the robot's jig and the parts it can handle.

[0040] The peripheral device DB132 is a peripheral device information storage unit that stores peripheral device information indicating the performance and price of each peripheral device for a plurality of candidate peripheral devices. The performance of the peripheral devices is information necessary when selecting one or more peripheral devices from work information, such as the shape and size of transportable parts, or the transportable route. Note that the peripheral devices may include a display device that visualizes the operating status, etc.

[0041] The robot selection unit 133 selects a robot from among multiple robots indicated in the robot information that can perform the work specified in the work information. For example, the robot selection unit 133 selects a robot that can perform the actions specified in the work information and that can realize the sequence of actions specified in the work information, the parts used in the actions and the positions for accessing those parts, and the gripping state indicating which part of the parts to grip, within the operating range and degrees of freedom. If, for example, the user has indicated in advance the budget that can be allocated to the robot, the robot selection unit 133 will select a robot priced at or below that budget.

[0042] The peripheral device selection unit 134 selects a peripheral device from among the multiple peripheral devices indicated in the peripheral device information that can perform the work content indicated in the work information.

[0043] For example, the peripheral device selection unit 134 selects a peripheral device that can transport the parts used in the operation indicated by the work information and can transport the parts to the position accessed in the operation indicated by the work information. In addition, when conditions regarding the performance or price of the display device to be used are indicated by the user in advance, for example, the peripheral device selection unit 134 also selects a display device that satisfies such conditions as a peripheral device. Further, when the cost that can be applied to the peripheral device is indicated by the user in advance, for example, the peripheral device selection unit 134 is assumed to select a peripheral device with a price below that cost.

[0044] The layout generation unit 135 determines the layout of the robot selected by the robot selection unit 133 and the peripheral device selected by the peripheral device selection unit 134, and generates design information for the robot system by specifying the operation paths of the robot and the peripheral device. The design information shall at least indicate the layout of the robot and the peripheral devices and the operation paths of the robot and the peripheral devices as a robot system. The generation of the design information may use, for example, a process performed by a known device such as a robot cell system design device disclosed in Japanese Patent Application Laid-Open No. 2022-134604.

[0045] In addition, when multiple robots are selected by the robot selection unit 133 or when multiple peripheral devices are selected by the peripheral device selection unit 134, in at least either one of these cases, the layout generation unit 135 may generate design information for each combination thereof.

[0046] The production simulator 136 is a simulation unit that executes a simulation of the productivity in the robot system indicated by the design information. For example, the production simulator 136 may use an existing product such as the 3D simulator "MELSOFT Gemini". The productivity here is, for example, the expected production volume.

[0047] The report generation unit 137 generates a proposal report that at least indicates the design information and the productivity that is the result of the simulation. The proposal report may show the price of the robot system. In addition, when a plurality of design information is generated, a proposal report may be created for each of the plurality of design information.

[0048] The robot system design device 100 described above can be realized by a computer such as the PC 10 shown in FIG. 4. The PC 10 includes a storage 11 such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive), a memory 12, a processor 13 such as a CPU (Central Processing Unit), a communication I / F (Interface) 14 such as a NIC (Network Interface Card), an input I / F 15 such as a keyboard and a mouse, and a display 16.

[0049] For example, the robot DB 131 and the peripheral device DB 132 can be realized by the storage 11 or the memory 12. The video acquisition unit 110, the work information generation unit 120, the robot selection unit 133, the peripheral device selection unit 134, the layout generation unit 135, the production simulator 136, and the report generation unit 137 can be realized by the processor 13 executing a program stored in the storage 11.

[0050] The program may be downloaded from a recording medium (not shown) via a reader / writer (not shown) or from a network via the communication I / F 14 to the storage 11, and then loaded onto the memory 12 and executed by the processor 13. Alternatively, the program may be directly loaded onto the memory 12 from a recording medium (not shown) via a reader / writer (not shown) or from a network via the communication I / F 14 and executed by the processor 13. In other words, the program may be provided by a computer program product such as a recording medium.

[0051] Furthermore, at least a portion of the video acquisition unit 110, work information generation unit 120, robot selection unit 133, peripheral device selection unit 134, layout generation unit 135, production simulator 136, and report generation unit 137 can also be composed of processing circuits such as single circuits, composite circuits, program-operated processors, program-operated parallel processors, ASICs (Application Specific Integrated Circuits), or FPGAs (Field Programmable Gate Arrays). As described above, the robot system design device 100 can be realized by a processing circuit network.

[0052] Figure 5 is a flowchart illustrating the operation of the video acquisition unit 110 and the work information generation unit 120. First, the video acquisition unit 110 acquires video data showing images of an worker performing a task (S10). The acquired video data is then provided to the work information generation unit 120.

[0053] Next, the 3D skeleton extraction unit 121 extracts 3D keypoint information from each frame contained in the video data, which indicates the 3D coordinates of keypoints, which are multiple feature points corresponding to multiple parts of the worker (S11).

[0054] Next, the object recognition unit 122 detects objects such as parts and tools from each frame included in the video data, recognizes the detected objects, and generates object information indicating those objects (S12).

[0055] Next, the object 3D pose identification unit 123 estimates the pose of the object recognized by the object recognition unit 122 for each frame and generates pose information indicating that pose (S13).

[0056] Next, the motion identification unit 124 shown in Figure 1 recognizes the worker's movements from 3D keypoint information, object information, and posture information (S14).

[0057] The work procedure identification unit 125 identifies a sequence of operations by arranging the operations recognized by the operation identification unit 124 in chronological order, and extracts a work procedure, which is an operation cycle for a single product, from that sequence of operations (S15).

[0058] Then, the work content identification unit 126 identifies the necessary information from 3D keypoint information, object information, and posture information corresponding to the frames from which the actions included in the work procedure extracted by the work procedure identification unit 125 have been extracted, thereby generating work information that identifies the worker's work content (S16). The generated work information is provided to the robot system design unit 130.

[0059] Figure 6 is a flowchart showing the operation of the robot system design unit 130. First, the robot selection unit 133 selects a robot capable of performing the tasks indicated by the work information from among a plurality of robots indicated by the robot information stored in the robot DB 131 (S20).

[0060] Next, the peripheral device selection unit 134 selects a peripheral device capable of executing the content indicated in the work information from among the multiple peripheral devices indicated by the peripheral device information stored in the peripheral device DB 132 (S21).

[0061] Next, the layout generation unit 135 generates design information for the robot system by laying out the robot selected by the robot selection unit 133 and the peripheral devices selected by the peripheral device selection unit 134, and by identifying the operating paths of the robot and its peripheral devices (S22).

[0062] Next, the production simulator 136 simulates the productivity of the robot system as shown in the design information (S23).

[0063] The report generation unit 137 then generates a proposal report that at least shows design information and productivity (S24). The report generation unit 137 may output the proposal report by, for example, displaying it on a display unit realized by the display 16 shown in Figure 4, or by transmitting it to another device connected to a network such as the Internet via a communication unit realized by the communication I / F 14 shown in Figure 4.

[0064] As described above, according to Embodiment 1, design information for the robot system can be automatically generated from video data captured of an operator performing a task, thereby reducing the burden on the user.

[0065] Embodiment 2. Figure 7 is a block diagram schematically showing the configuration of the robot system design apparatus 200 according to Embodiment 2. The robot system design apparatus 200 comprises an image acquisition unit 110, a work information generation unit 120, and a robot system design unit 230.

[0066] The video acquisition unit 110 and work information generation unit 120 of the robot system design apparatus 200 according to Embodiment 2 are the same as the video acquisition unit 110 and work information generation unit 120 of the robot system design apparatus 100 according to Embodiment 1.

[0067] The robot system design unit 230 designs the robot system from the work information generated by the work information generation unit 120. The robot system design unit 230 includes a robot DB 131, a peripheral device DB 132, a robot selection unit 133, a peripheral device selection unit 134, a layout generation unit 135, a production simulator 136, a report generation unit 237, and a case study database (hereinafter referred to as the case study DB) 238.

[0068] In Embodiment 2, the robot DB 131, peripheral device DB 132, robot selection unit 133, peripheral device selection unit 134, layout generation unit 135, and production simulator 136 of the robot system design unit 230 are the same as those of the robot DB 131, peripheral device DB 132, robot selection unit 133, peripheral device selection unit 134, layout generation unit 135, and production simulator 136 of the robot system design unit 130 in Embodiment 1.

[0069] The report generation unit 237 generates a proposal report that at least shows design information and productivity, similar to the report generation unit 137 in Embodiment 1. The report generation unit 237 then stores the generated proposal report in the case study database 238.

[0070] Furthermore, the report generation unit 237 may output the proposed report by, for example, displaying it on a display unit realized by the display 16 shown in Figure 4, or by transmitting it to another device connected to a network such as the Internet via a communication unit realized by the communication I / F 14 shown in Figure 4.

[0071] Here, when the report generation unit 237 outputs the proposal report that has been generated, it may select past proposal reports related to the current proposal report and output the selected proposal reports together. The related past proposal reports shall satisfy predetermined conditions such as the robot being used is the same, at least some of the parts being used is the same, at least some of the operation paths being the same, or the products produced by the work being the same.

[0072] Case DB238 is a storage unit that stores proposal reports generated in the past.

[0073] The robot system design apparatus 200 described above can also be implemented using a computer such as the PC 10 shown in Figure 4. For example, the example DB 238 can also be implemented using storage 11 or memory 12. The robot system design apparatus 200 can also be implemented using a processing circuit network.

[0074] As described above, according to Embodiment 2, when outputting a proposal report, related past proposal reports can also be output, allowing the user to refer to these proposal reports and consider the robot system.

[0075] 100, 200 Robot system design device, 110 Image acquisition unit, 120 Work information generation unit, 121 3D skeleton extraction unit, 122 Object recognition unit, 123 Object 3D posture identification unit, 124 Motion identification unit, 125 Work procedure identification unit, 126 Work content identification unit, 130, 230 Robot system design unit, 131 Robot DB, 132 Peripheral device DB, 133 Robot selection unit, 134 Peripheral device selection unit, 135 Layout generation unit, 136 Production simulator, 137, 237 Report generation unit, 238 Case study DB.

Claims

1. A robot system design device comprising: an object recognition unit that recognizes an object from video footage of an operator repeatedly performing a task; an orientation identification unit that identifies the orientation of the object from the video footage; a skeleton extraction unit that extracts the skeleton of the operator from the video footage; an action identification unit that identifies multiple actions of the operator from the object, the orientation, and the skeleton; an action procedure identification unit that identifies a work procedure, which is the sequence of the task, based on two or more actions that are repeatedly performed in the multiple actions; an action content identification unit that identifies the content of the task, which is the content of the task, based on the work procedure and the object, the orientation, and the skeleton at the time of the work procedure; and a robot system design unit that generates design information for a robot system that indicates at least a robot and peripheral devices capable of realizing the action content, and the operation paths of the robot and peripheral devices.

2. The robot system design apparatus according to claim 1, characterized in that the motion identification unit identifies the plurality of motions based on the time series of the positions of predetermined parts of the worker included in the skeleton and the positions of the objects, or the time series of the positions of the objects themselves.

3. The robot system design apparatus according to claim 1, characterized in that the motion identification unit identifies the plurality of motions based on the time series of the positions of predetermined parts of the worker included in the skeleton that have predetermined interactions with each other, and the positions of the objects, or the time series of the positions of the objects themselves.

4. The robot system design apparatus according to any one of claims 1 to 3, characterized in that the robot system design unit comprises: a robot selection unit that selects a robot capable of performing the work from a predetermined plurality of robots; a peripheral device selection unit that selects a peripheral device capable of performing the work from a predetermined plurality of peripheral devices; and a layout generation unit that generates the design information by determining the layout of the robot and the peripheral device and specifying the operation path of the robot and the peripheral device.

5. The robot system design apparatus according to claim 4, further comprising: a simulation unit that performs a simulation of the productivity of the robot system based on the design information; and a report generation unit that generates a proposal report including the design information and the results of the simulation.

6. The robot system design apparatus according to claim 5, further comprising a storage unit for storing proposal reports generated in the past, wherein the report generation unit selects one or more proposal reports related to the currently generated proposal report from the previously generated proposal reports and outputs the one or more proposal reports together with the currently generated proposal report.

7. A program characterized in that it causes a computer to function as: an object recognition unit that recognizes an object from video footage of an operator repeatedly performing a task; an orientation identification unit that identifies the orientation of the object from the video footage; a skeleton extraction unit that extracts the skeleton of the operator from the video footage; an action identification unit that identifies multiple actions of the operator from the object, the orientation, and the skeleton; an action procedure identification unit that identifies an action procedure, which is the sequence of the task, based on two or more actions that are repeatedly performed in the multiple actions; an action content identification unit that identifies an action content, which is the content of the task, based on the action procedure and the object, the orientation, and the skeleton at the time of the action procedure; and a robot system design unit that generates design information for a robot system that shows at least a robot and peripheral devices capable of realizing the action content, and the action paths of the robot and peripheral devices.

8. A robot system design method characterized by: recognizing an object from video footage of an operator repeatedly performing a task; identifying the posture of the object from the video; extracting the operator's skeleton from the video; identifying multiple actions of the operator from the object, posture and skeleton; identifying a work procedure, which is the sequence of the task, based on two or more actions that are repeatedly performed in the multiple actions; identifying the content of the task, which is the content of the task, based on the work procedure and the object, posture and skeleton at the time of the work procedure; and generating design information for a robot system that shows at least a robot and peripheral devices capable of realizing the content of the task, and the operating paths of the robot and peripheral devices.

Images