Screw detecting system, screw fastening system equipped with same, and screw detecting method

Abstract

This screw detecting system comprises a three-dimensional sensor and a processing circuit, wherein the processing circuit: generates, from distance data, a brightness distribution image of a male screw having different brightnesses depending on distance; generates a second-order differential image obtained by taking the second-order differential of the brightness distribution image; generates a third-order differential image obtained by taking the third-order differential of the brightness distribution image; calculates the logical product of the second-order differential image and the third-order differential image to generate an edge extraction image indicating the brightness distribution on the thread ridges of the male screw; converts the brightness values in the edge extraction image into three-dimensional point cloud data for distance; executes cylindrical fitting processing for fitting a cylindrical model having a preset outer diameter to the three-dimensional point cloud data; and determines the position and orientation of the central axis of the cylindrical model after execution of the cylindrical fitting processing as the position and orientation of the central axis of the male screw.

Description

Screw detection system, screw fastening system including the same, and screw detection method

[0001] The present disclosure relates to a screw detection system, a screw fastening system including the same, and a screw detection method.

[0002] When causing an automatic machine such as a robot to perform an operation of fastening a fixture such as a nut to a male screw disposed on a workpiece, if the position of the male screw is not accurate with respect to the teaching position of the automatic machine, an appropriate fastening operation cannot be realized. On the other hand, when the workpiece is installed on a workbench or the like, an installation error may occur. In this case, the position of the male screw disposed on the workpiece deviates from the position assumed from the teaching position of the automatic machine. Further, even if the workpiece is installed at an appropriate position due to dimensional errors or the like of the workpiece, the position of the male screw may deviate from the position assumed from the teaching position of the automatic machine.

[0003] Patent Document 1 below discloses automatically measuring the dimensions of a male screw using an optical measuring device.

[0004] Japanese Patent Application Laid-Open No. 2016-27320

[0005] In the method of Patent Document 1, the male screw is installed at an appropriate position as a premise of measurement. For this reason, in the method of Patent Document 1, the male screw is measured from above in the axial direction of the male screw and in a direction perpendicular to the axial direction. However, for example, when the male screw is disposed in a narrow portion of the workpiece or the like, depending on the location of the male screw on the workpiece, there are cases where measurement cannot be performed from a fixed direction. Further, the male screw is not always disposed perpendicular to the workpiece.

[0006] Therefore, there is room for improvement in applying the method of Patent Document 1 as a system for detecting the position and orientation of a male screw such that the actual position of the male screw can deviate from the assumed position.

[0007] The present disclosure has been made in view of the above problems, and an object thereof is to provide a screw detection system capable of appropriately detecting the position and orientation of a male screw, a screw fastening system including the same, and a screw detection method.

[0008] A screw detection system according to one aspect of the present disclosure is a screw detection system for detecting the position and orientation of a male screw, comprising: a three-dimensional sensor positioned at a predetermined detection position based on teaching data for the position and orientation of the central axis of the male screw, and measuring the distance to the male screw; and a processing circuit that processes the distance data from the detection position to the male screw measured by the three-dimensional sensor, wherein the processing circuit generates a brightness distribution image of the male screw, where the brightness differs according to the distance from the detection position to the male screw, from the distance data, and generates a second derivative image obtained by taking the second derivative of the brightness distribution image. The process involves generating a third-order derivative image by taking the third derivative of the luminance distribution image, generating an edge extraction image showing the luminance distribution at the top of the threads of the male screw as a result of calculating the logical AND of the second-order derivative image and the third-order derivative image, converting the luminance values ​​in the edge extraction image into three-dimensional point cloud data for distance, performing a cylinder fitting process to fit a cylinder model having a predetermined outer diameter from the outer diameter data of the male screw to the three-dimensional point cloud data, and determining the position and orientation of the central axis of the cylinder model after the execution of the cylinder fitting process as the position and orientation of the central axis of the male screw.

[0009] A screw fastening system according to another aspect of the present disclosure comprises: the screw detection system; a fastening jig for holding a fixed object having a female thread corresponding to the male screw and fastening the fixed object to the male screw; a manipulator arm for selectively holding the fastening jig and the three-dimensional sensor at its tip and for changing the position and orientation of the tip in three dimensions; and a control circuit for controlling the manipulator arm, wherein the control circuit positions the three-dimensional sensor at the detection position while holding the three-dimensional sensor at the tip of the manipulator arm; compares the position and orientation of the central axis of the male screw obtained based on the distance data acquired by the three-dimensional sensor at the detection position with the position and orientation in the teaching data to correct the teaching position at the tip of the manipulator arm for fastening the fixed object to the male screw; moves the manipulator arm to the corrected teaching position while holding the fastening jig at the tip of the manipulator arm and fastens the fixed object to the male screw using the fastening jig.

[0010] Another aspect of the present disclosure is a screw detection method for detecting the position and orientation of a male screw, comprising: measuring the distance from a predetermined detection position to the male screw based on teaching data for the position and orientation of the central axis of the male screw that has been set in advance; generating a brightness distribution image of the male screw, in which the brightness differs according to the distance from the detection position to the male screw, from the distance data obtained as a result of the measurement; generating a second derivative image by taking the second derivative of the brightness distribution image; generating a third derivative image by taking the third derivative of the brightness distribution image; generating an edge extraction image showing the brightness distribution at the top of the threads of the male screw as a result of calculating the logical AND of the second derivative image and the third derivative image; converting the brightness values ​​in the edge extraction image into three-dimensional point cloud data for distance; performing a cylinder fitting process to fit a cylinder model having a predetermined outer diameter from the external shape data of the male screw to the three-dimensional point cloud data; and determining the position and orientation of the central axis of the cylinder model after the execution of the cylinder fitting process as the position and orientation of the central axis of the male screw.

[0011] According to this disclosure, the position and orientation of the male screw can be appropriately detected.

[0012] Figure 1 is a diagram showing the schematic configuration of a screw detection system according to one embodiment of the present disclosure. Figure 2 is an image diagram showing the flow of image processing based on distance data in this embodiment. Figure 3 is a diagram showing the positional relationship between a portion of a luminance distribution image obtained in this embodiment and the threads of a male screw. Figure 4 is an image diagram for extracting the tops of the threads of a male screw from a graph showing the luminance distribution in a portion of the luminance distribution image shown in Figure 3. Figure 5 is a diagram showing the positional relationship between a portion of a luminance distribution image obtained in this embodiment and the threads of a male screw in another example. Figure 6 is an image diagram for extracting the tops of the threads of a male screw from a graph showing the luminance distribution in a portion of the luminance distribution image shown in Figure 5. Figure 7 is a diagram showing the schematic configuration of a screw fastening system in this embodiment.

[0013] An embodiment will be described in detail below with reference to the drawings. In the following, the same or corresponding elements will be denoted by the same reference numerals throughout all the drawings, and redundant explanations will be omitted.

[0014] Figure 1 is a diagram showing a schematic configuration of a screw detection system according to one embodiment of the present disclosure. The screw detection system 1 comprises a three-dimensional sensor 2 and a processing circuit 3. In this embodiment, the three-dimensional sensor 2 is held at the tip of the manipulator arm 12 of a robot 11 that constitutes a screw fastening system 10, which will be described later. The three-dimensional sensor 2 is a sensor that measures the distance to an object, and various sensors such as LiDAR (light detection and ranging) can be used.

[0015] The manipulator arm 12 has multiple joints 5. The multiple joints 5 constitute the motion axes for changing the position and orientation of the 3D sensor 2 held at the tip of the manipulator arm 12. In the example in Figure 1, a manipulator arm 12 having six joints 5 is shown, but the number of joints 5 may be seven or more, or five or less. The manipulator arm 12 is fixed to a base 7. The base 7 may be fixed to a fixed object such as the ground, or it may be configured to be movable. A tool changer 8 is fixed to the tip of the manipulator arm 12. The 3D sensor 2 is detachably held by the tool changer 8.

[0016] Multiple motion axes constituting multiple joints 5 are driven by multiple robot drive motors. The robot 11 is equipped with a robot controller 9. The multiple robot drive motors are controlled by the robot controller 9. This allows the manipulator arm 12 to change the position and orientation of its tip in three dimensions. The robot controller 9 generates robot control command values ​​for driving the robot drive motors. The robot controller 9 is equipped with a robot control circuit that generates robot control command values ​​for controlling the robot drive motors. The robot control circuit has a computer such as a microcontroller or a PLC (Programmable Logic Controller).

[0017] The robot controller 9 is connected to the higher-level controller 13. The higher-level controller 13 acquires the operating status of the robot controller 9 and outputs control commands to the robot controller 9. The higher-level controller 13 is equipped with a control circuit 14 that performs various signal processing. The control circuit 14 generates control commands to control the manipulator arm 12.

[0018] The processing circuit 3 is connected to the 3D sensor 2 and processes the data measured by the 3D sensor 2. The processing circuit 3 may be located inside the housing of the 3D sensor 2, or it may be configured separately from the 3D sensor 2. The processing circuit 3 is connected to the control circuit 14. Alternatively, the processing circuit 3 may be configured as the control circuit 14 of the higher-level controller 13. Furthermore, the processing circuit 3 may consist of multiple circuits. For example, a processing circuit such as a server connected to the 3D sensor 2 may perform some of the processing performed by the processing circuit 3 as described later, and the remaining processing may be performed by the processing circuit inside the housing of the 3D sensor 2 or by the control circuit 14.

[0019] The processing circuit 3 and the control circuit 14 include a computer such as a microcontroller, personal computer, or PLC (Programmable Logic Controller). More specifically, the processing circuit 3 and the control circuit 14 each include a processor, memory, and peripheral circuits. The processor includes, for example, a CPU or MPU. The memory includes ROM, RAM, registers, non-volatile storage, etc. The peripheral circuits include input / output interfaces, etc. The higher-level controller 13 may include an output device such as a monitor for displaying output or a speaker for outputting sound, or an input device for user operation input.

[0020] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, dedicated processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuits. In this specification, a circuit, unit, means, or part is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. The hardware may be hardware disclosed herein, or other known hardware that is programmed or configured to perform the enumerated functions. If the hardware is a processor, which is considered a type of circuit, then the circuit, unit, or means is a combination of hardware and software, and the software is used to configure the hardware and / or the processor.

[0021] In this embodiment, the screw detection system 1 is used to detect the position and orientation of a male screw T fixed to a predetermined location on the workpiece W. The workpiece W includes a base surface W1 installed on a workpiece table B, a wall surface W2 rising vertically from the base surface W1, and an upper surface W3 extending parallel to the base surface W1 from the upper end of the wall surface W2.

[0022] The position in which the workpiece W is placed on the work table B is predetermined. As will be described later, the teaching data indicating the position and orientation of the tip of the manipulator arm 12 for fastening the fixed device to the male screw T by the screw fastening system 10 is predetermined based on the position and orientation of the male screw T when the workpiece W is set to a predetermined position on the work table B. However, due to installation errors of the workpiece W or dimensional errors of the workpiece W, the actual position and orientation of the male screw T may deviate from the position and orientation of the male screw T corresponding to the taught position. The screw detection system 1 identifies the actual position and orientation of the male screw T in order to correct this positional deviation.

[0023] To this end, the control circuit 14 first controls the manipulator arm 12 so that the tip position of the manipulator arm 12 holding the 3D sensor 2 is located at the detection position. The detection position is predetermined based on the teaching data of the male screw T. As a result, the 3D sensor 2 is positioned at the detection position such that the optical axis C1 has a predetermined inclination angle θ with respect to the central axis C2 of the male screw T in the teaching data. In the example of Figure 1, the central axis C2 of the male screw T is positioned perpendicular to the base surface W1 of the workpiece W. That is, the 3D sensor 2 is positioned at the detection position such that the optical axis C1 is inclined with respect to the base surface W1, i.e., the surface of the workpiece table B.

[0024] The processing circuit 3 pre-determines the correlation between the coordinate system of the image captured by the 3D sensor 2 and the coordinate system in real space. The processing circuit 3 may determine the correlation from the real-space position coordinates of the detection position where the 3D sensor 2 is placed and the orientation of the 3D sensor 2, or it may determine the correlation by detecting a predetermined calibration target placed at a specific position with the 3D sensor 2. The determined correlation is stored in the memory of the processing circuit 3.

[0025] The 3D sensor 2 measures the distance from the detection position to the male thread T of the workpiece W, which is placed at a predetermined position on the workpiece table B. The distance data from the detection position to the male thread T obtained by the measurement is sent to the processing circuit 3 and stored in the memory. The processing circuit 3 performs the following processing on the acquired distance data.

[0026] Figure 2 is an illustrative diagram showing the flow of image processing based on distance data in this embodiment. The processing circuit 3 generates a brightness distribution image G1 of male screws T, where the brightness differs depending on the distance from the detection position to the male screw T, from the distance data. The brightness distribution image G1 is generated as an image where, for example, the higher the brightness, the closer the distance and the lower the brightness, the farther the distance. The brightness distribution image G1 is composed of multiple pixels, and each pixel has a brightness value. That is, the brightness distribution image G1 has distance information for each pixel.

[0027] Processing circuit 3 generates a second derivative image G2 by taking the second derivative of the luminance distribution image G1. Furthermore, processing circuit 3 generates a third derivative image G3 by taking the third derivative of the luminance distribution image G1. Processing circuit 3 performs a logical AND operation on the second derivative image G2 and the third derivative image G3, and as a result generates an edge extraction image G4 that shows the luminance distribution at the top of the threads of the male screw T.

[0028] Figure 3 shows the positional relationship between a portion of a luminance distribution image obtained in this embodiment and the threads of a male screw. The top figure in Figure 3 is a schematic diagram of the threads M of the male screw T. The middle figure in Figure 3 shows a portion of the luminance distribution image G1p (G1p). The bottom graph in Figure 3 shows the luminance distribution in the portion G1p. For ease of understanding, the displacement of the threads M in the direction of the central axis C2 is ignored in Figure 3.

[0029] As mentioned above, the optical axis C1 of the 3D sensor 2 is inclined at an angle θ with respect to the central axis C2 of the male screw T. Therefore, the region of the screw threads M of the male screw T that is on the opposite side of the screw thread climax Mt from the 3D sensor 2 becomes a blind spot for the 3D sensor 2. For this reason, the low-luminance region of the luminance distribution image G1 includes a part of the screw threads M and a portion without screw threads M, i.e., the valley V, as shown by the shaded region in Figure 3.

[0030] Therefore, as shown in the brightness distribution graph, in the brightness distribution image G1, the boundary between the thread crest Mt of the male screw T and the blind spot region D shows a large brightness difference. On the other hand, the boundary between the thread crest M and the valley V of the male screw T shows a relatively small brightness difference. Here, the second derivative image G2 shows the degree of brightness change in the direction perpendicular to the ridge of the thread crest Mt, that is, in the direction along the gradient, in the brightness distribution image G1. Therefore, in the second derivative image G2, the second derivative value is 0 at the position of the thread crest Mt, and the second derivative value increases sharply with a positive value as you move from the thread crest Mt towards the blind spot region D, while the second derivative value decreases with a negative value as you move from the thread crest Mt towards the region facing the 3D sensor 2.

[0031] Furthermore, the third-order derivative image G3 roughly traces the change in brightness distribution. Therefore, the third-order derivative image G3 can be represented as a cubic function along the central axis C2 direction, where the third derivative value is highest near the top of the screw thread Mt and lowest near the valley V. Consequently, the change in brightness is zero at the highest value of the third-order derivative image G3, and zero at the lowest value of the third derivative.

[0032] Based on the characteristics of the second-order derivative image G2 and the third-order derivative image G3, an edge extraction image G4 is generated by extracting the thread peaks Mt of the male screw T. To this end, the processing circuit 3 generates a first extraction image by extracting pixels from among the multiple pixels constituting the second-order derivative image G2 whose second derivative value is between a predetermined lower limit less than 0 and 0. The processing circuit 3 also generates a second extraction image by extracting pixels from among the multiple pixels constituting the third-order derivative image G3 whose change in the third derivative value is within a predetermined range including 0.

[0033] Figure 4 is an illustrative diagram for extracting the thread peaks of a male screw from a graph showing the brightness distribution in a portion of the brightness distribution image shown in Figure 3. In the upper graph of Figure 4, the oval area represents the region of pixels among the multiple pixels constituting the second derivative image G2 where the second derivative value is between a predetermined lower limit less than 0 and 0. In the middle graph of Figure 4, the circular area represents the region of pixels among the multiple pixels constituting the third derivative image G3 where the change in the third derivative value is within a predetermined range including 0.

[0034] Furthermore, the shaded circle in the lower graph of Figure 4 indicates the region of pixels extracted as a result of taking the logical AND of the first extracted image extracted from the second differential image G2 and the second extracted image extracted from the third differential image G3. Thus, the edge extraction image G4 obtained as a result of the logical AND of the second differential image G2 and the third differential image G3 is an image that extracts the vicinity of the edge of the thread crest Mt on the male screw T.

[0035] Furthermore, depending on the orientation of the 3D sensor 2 at the detection position, the horizontal axis of the graph showing the brightness distribution may be inverted. That is, when the 3D sensor 2 is positioned at the detection position, it is conceivable that the area above a predetermined origin is positive, or that the area above the central axis is negative.

[0036] Figure 5 shows the positional relationship between a portion of the brightness distribution image obtained in this embodiment and the threads of the male screw. The graph shown in Figure 5 is equivalent to the graph shown in Figure 3 with the horizontal axis reversed. As a result, in the second derivative image G2, the second derivative value is 0 at the position of the thread crest Mt, and the second derivative value increases sharply as it moves from the thread crest Mt towards the blind spot region D, while the second derivative value increases sharply as it moves from the thread crest Mt towards the region facing the 3D sensor 2, with the second derivative value increasing sharply.

[0037] Figure 6 is an illustrative diagram for extracting the thread peaks of a male screw from a graph showing the brightness distribution in a portion of the brightness distribution image shown in Figure 5. Due to the inversion of the horizontal axis, the oval portion in the upper graph of Figure 6 represents the region of pixels among the multiple pixels constituting the second derivative image G2 whose second derivative value is between 0 and a predetermined upper limit greater than 0. In this example, the processing circuit 3 generates a first extracted image by extracting pixels among the multiple pixels constituting the second derivative image G2 whose second derivative value is between 0 and a predetermined upper limit greater than 0. The processing circuit 3 generates a second extracted image, similar to the embodiments in the examples of Figures 3 and 4, and generates an edge extracted image G4 by performing a logical AND operation between the first extracted image and the second extracted image.

[0038] The processing circuit 3 converts the luminance values ​​in the generated edge extraction image G4 into three-dimensional point cloud data representing distance. In this embodiment, since the original luminance distribution image G1 is generated as an image where higher luminance corresponds to a closer distance and lower luminance corresponds to a farther distance, the processing circuit 3 converts the luminance value data of each pixel in the edge extraction image G4 into pixel-specific distance data such that higher luminance corresponds to a closer distance and lower luminance corresponds to a farther distance.

[0039] The processing circuit 3 performs a cylinder fitting process to fit a predetermined cylinder model to the converted 3D point cloud data. Image G5 in Figure 2 shows the state after fitting the 3D point cloud data to the cylinder model. The outer diameter of the cylinder model is set in advance from the outer diameter data of the male screw T. For example, the outer diameter of the cylinder model may be the same as the outer diameter of the male screw T, or it may be an outer diameter that is larger or smaller by a predetermined error range.

[0040] The processing circuit 3 identifies the central axis C2 of the male screw T from the results of the cylindrical fitting process. More specifically, the processing circuit 3 converts the position and orientation of the central axis of the cylindrical model after the cylindrical fitting process into a real-space coordinate system. The processing circuit 3 determines the position and orientation of the central axis of the cylindrical model in the real-space coordinate system as the position and orientation of the central axis C2 of the male screw T. The processing circuit 3 transmits the identified position and orientation data of the central axis C2 of the male screw T as real data to the control circuit 14 of the higher-level controller 13. The control circuit 14 stores the real data in a memory.

[0041] Figure 7 shows a schematic configuration of the screw fastening system in this embodiment. As described above, the screw fastening system 10 in this embodiment includes a robot 11 having the manipulator arm 12, a robot controller 9, and a higher-level controller 13. In other words, the screw fastening system 10 and the screw detection system 1 share the robot 11, the robot controller 9, and the higher-level controller 13.

[0042] The screw fastening system 10 includes a fastening jig 15 fixed to the tip of the manipulator arm 12, instead of the three-dimensional sensor 2 in the screw detection system 1. The fastening jig 15 is detachably held by the tool changer 8. In this way, the manipulator arm 12 selectively holds the fastening jig 15 and the three-dimensional sensor 2 by the tool changer 8 located at the tip of the manipulator arm 12.

[0043] The fastening jig 15 holds the fixture U having a female screw corresponding to the male screw T, and is a jig for fastening the fixture U to the male screw T. For example, the fastening jig 15 is a nut runner or the like. The fixture U includes, for example, a nut or the like. The fixture U may be another workpiece fixed to the workpiece W via the male screw T. In this case, the fastening jig 15 may be a robot hand, a suction device, or the like for gripping the workpiece.

[0044] As described above, the control circuit 14 positions the three-dimensional sensor 2 at the detection position while holding the three-dimensional sensor 2 at the tip of the manipulator arm 12, and causes the three-dimensional sensor 2 to measure the male screw T. The control circuit 14 acquires the position and orientation of the central axis C2 of the male screw T in the actual data obtained based on the distance data acquired by the three-dimensional sensor 2 at the detection position. The control circuit 14 compares the position and orientation of the central axis C2 of the male screw T in the acquired actual data with the position and orientation in the teaching data stored in the memory. Based on the comparison result, the control circuit 14 corrects the teaching position at the tip of the manipulator arm 12 for fastening the fixture U to the male screw T.

[0045] The three-dimensional sensor 2 held at the tip of the manipulator arm 12 is removed, and instead, the fastening jig 15 is held. The replacement from the three-dimensional sensor 2 to the fastening jig 15 may be performed manually by an operator or automatically by the robot 11 based on a control command of the control circuit 14. The control circuit 14 moves the manipulator arm 12 to the corrected teaching position while holding the fastening jig 15 at the tip of the manipulator arm 12, and performs operation control to fasten the fixture U to the male screw T using the fastening jig 15.

[0046] According to this embodiment, the distance from the detection position to the male screw T is measured using the three-dimensional sensor 2 arranged at a predetermined detection position. Further, a logical product of the second derivative image G2 obtained by second-differentiating the luminance distribution image G1 generated based on the measurement result and the third derivative image G3 obtained by third-differentiating the luminance distribution image G1 is calculated. Thereby, even if the optical axis C1 of the three-dimensional sensor 2 is not orthogonal to the central axis C2 of the male screw T, an edge extraction image G4 at the thread crest Mt of the male screw T can be generated. Further, by performing fitting between the three-dimensional point cloud data regarding the distance based on the edge extraction image G4 and a predetermined cylindrical model, the position and orientation of the central axis C2 of the male screw T in the real space can be specified. Therefore, regardless of the detection position where the three-dimensional sensor 2 is arranged, the position and orientation of the actual male screw T can be appropriately detected.

[0047] As a result, since the three-dimensional sensor 2 can be arranged such that the optical axis C1 is inclined with respect to the central axis C2 of the male screw T, even for the workpiece W in which the male screw T is arranged in the narrow part, the position and orientation of the male screw T can be appropriately detected. Also, the degree of freedom in setting the detection position of the three-dimensional sensor 2 can be increased.

[0048] Furthermore, according to this embodiment, the three-dimensional sensor 2 is held at the tip of the manipulator arm 12 having a plurality of joints and capable of three-dimensionally changing the position and orientation of the tip. Thereby, the three-dimensional sensor 2 can be easily held in a desired posture at the detection position. Also, since the position and orientation of the tip of the manipulator arm 12 can be changed three-dimensionally, the degree of freedom in setting the detection position of the three-dimensional sensor 2 can be increased.

[0049] Furthermore, according to this embodiment, before calculating the logical AND, pixels between 0 and a predetermined lower limit of the second derivative value, or between 0 and a predetermined upper limit of the second derivative value, or between 0 and a predetermined upper limit of the second derivative value, are extracted from the second derivative image G2, thereby enabling the extraction of edges near the thread crest Mt of the male screw T. Moreover, before calculating the logical AND, pixels within a predetermined range including 0 are extracted from the third derivative image G3, thereby enabling the extraction of the ridge line between the thread crest Mt and the valley V of the male screw T. As a result, when calculating the logical AND, edge extraction of the thread crest Mt of the male screw T can be appropriately performed.

[0050] Furthermore, according to this embodiment, after detecting the position and orientation of the male screw T using the three-dimensional sensor 2 held at the tip of the manipulator arm 12, the fastening operation using the manipulator arm 12 is performed by holding a fastening jig 15 for fastening the fixed device U to the male screw T at the tip of the manipulator arm 12 instead of the three-dimensional sensor 2. Therefore, the structure for positioning the three-dimensional sensor 2 at the detection position and the structure for performing the fastening operation can be made common. Consequently, it is not necessary to prepare a screw detection system 1 for detecting the position and orientation of the male screw T independently of the screw fastening system 10.

[0051] Furthermore, the teaching position at the tip of the manipulator arm 12 for fastening the fastener U to the male screw T is corrected according to the position and orientation of the male screw T identified by the screw detection system 1. This increases the success rate of fastening the fastener U to the male screw T in the screw fastening system 10.

[0052] While embodiments of this disclosure have been described above, this disclosure is not limited to the embodiments described above, and various improvements, changes, and modifications are possible without departing from the spirit of this disclosure.

[0053] [Other Embodiments] For example, in the above embodiment, an example was given in which the position and orientation of the male screw T fixed to the workpiece W is identified by the screw detection system 1 in order to correct the teaching position of the screw fastening system 10 that fastens a fastening device U to the male screw T fixed to the workpiece W. However, the use of the position and orientation of the male screw T identified by the screw detection system 1 is not limited to this. For example, the position and orientation of the male screw T identified by the screw detection system 1 may be used to correct the teaching position of a robot system that performs the operation of tightening a screw structure having a male screw T, such as a bolt, on a workpiece W that has been temporarily fastened with the male screw T.

[0054] Alternatively, in a picking system that uses a robotic hand to pick up screw structures such as bolts having male threads T placed inside a container, the position and orientation of the male threads T identified by the screw detection system 1 can also be utilized. In other words, by appropriately detecting the position and orientation of the male threads T to be picked in the picking system, the success rate of picking can be increased.

[0055] Furthermore, although the above embodiment illustrates a configuration in which some components, such as the manipulator arm 12, are shared between the screw detection system 1 and the screw fastening system 10, these systems 1 and 10 may be configured independently of each other.

[0056] [Summary of this Disclosure] [Item 1] A screw detection system according to one aspect of this Disclosure is a screw detection system for detecting the position and orientation of a male screw, comprising: a three-dimensional sensor positioned at a predetermined detection position based on teaching data for the position and orientation of the central axis of the male screw, and measuring the distance to the male screw; and a processing circuit that processes the distance data from the detection position to the male screw measured by the three-dimensional sensor, wherein the processing circuit generates a brightness distribution image of the male screw, where the brightness differs according to the distance from the detection position to the male screw, from the distance data, and generates a second derivative image obtained by taking the second derivative of the brightness distribution image. The process involves generating a third-order derivative image by taking the third derivative of the luminance distribution image, generating an edge extraction image showing the luminance distribution at the top of the threads of the male screw as a result of calculating the logical AND of the second-order derivative image and the third-order derivative image, converting the luminance values ​​in the edge extraction image into three-dimensional point cloud data for distance, performing a cylinder fitting process to fit a cylinder model having a predetermined outer diameter from the outer diameter data of the male screw to the three-dimensional point cloud data, and determining the position and orientation of the central axis of the cylinder model after the execution of the cylinder fitting process as the position and orientation of the central axis of the male screw.

[0057] According to the above configuration, the distance from the detection position to the male screw is measured using a 3D sensor placed at a predetermined detection position. Furthermore, a logical AND operation is performed between the second derivative image obtained by taking the second derivative of the brightness distribution image generated based on the measurement result and the third derivative image obtained by taking the third derivative of the brightness distribution image. This makes it possible to generate edge extraction images at the top of the threads of the male screw even if the optical axis of the 3D sensor is not orthogonal to the central axis of the male screw. Furthermore, by fitting the 3D point cloud data for distance based on the edge extraction image with a predetermined cylindrical model, the position and orientation of the central axis of the male screw in real space can be determined. Therefore, the actual position and orientation of the male screw can be appropriately detected regardless of the detection position where the 3D sensor is placed.

[0058] [Item 2] In the screw detection system of Item 1, the three-dimensional sensor may be positioned such that, at the detection position, its optical axis has a predetermined inclination angle with respect to the central axis of the male screw in the teaching data. By positioning the three-dimensional sensor so that its optical axis is inclined with respect to the central axis of the male screw, the position and orientation of the male screw can be appropriately detected even in workpieces where the male screw T is located in a narrow space. Furthermore, the degree of freedom in setting the detection position of the three-dimensional sensor can be increased.

[0059] [Item 3] The screw detection system of Item 1 or 2 may include a manipulator arm that holds the three-dimensional sensor at its tip and whose position and orientation can be changed in three dimensions. This makes it easy to hold the three-dimensional sensor in a desired orientation at the detection position. Furthermore, since the position and orientation of the tip of the manipulator arm can be changed in three dimensions, the degree of freedom in setting the detection position of the three-dimensional sensor can be increased.

[0060] [Item 4] In any of the screw detection systems of Items 1 to 3, the processing circuit may generate a first extracted image by extracting pixels from a plurality of pixels constituting the second derivative image whose second derivative value is between a predetermined lower limit less than 0 and 0, or between 0 and a predetermined upper limit greater than 0; generate a second extracted image by extracting pixels from a plurality of pixels constituting the third derivative image whose change in the third derivative value is within a predetermined range including 0; and generate the edge extraction image by performing a logical AND operation between the first extracted image and the second extracted image. According to this, before performing the logical AND operation, pixels whose second derivative value is between 0 and a predetermined lower limit less than 0, or between 0 and a predetermined upper limit greater than 0, are extracted from the second derivative image, thereby enabling the extraction of edges near the top of the threads of the male screw. Furthermore, before performing the logical AND operation, pixels whose change in the third derivative value is within a predetermined range including 0 are extracted from the third derivative image, thereby enabling the extraction of ridges between the top and bottom of the threads of the male screw. Therefore, when performing a logical AND operation, edge extraction at the thread peaks of the male screw T can be performed appropriately.

[0061] [Item 5] A screw fastening system according to another aspect of the present disclosure comprises a screw detection system according to any of items 1 to 4; a fastening jig for holding a fixed object having a female thread corresponding to the male thread and fastening the fixed object to the male thread; a manipulator arm for selectively holding the fastening jig and the three-dimensional sensor at its tip and for changing the position and orientation of the tip in three dimensions; and a control circuit for controlling the manipulator arm, wherein the control circuit positions the three-dimensional sensor at the detection position with the three-dimensional sensor held at the tip of the manipulator arm, compares the position and orientation of the central axis of the male thread obtained based on the distance data acquired by the three-dimensional sensor at the detection position with the position and orientation in the teaching data to correct the teaching position at the tip of the manipulator arm for fastening the fixed object to the male thread, moves the manipulator arm to the corrected teaching position with the fastening jig held at the tip of the manipulator arm, and fastens the fixed object to the male thread using the fastening jig.

[0062] According to the above configuration, the position and orientation of the male screw are detected using a 3D sensor held at the tip of the manipulator arm. Then, a fastening jig for fastening the fastener to the male screw is held at the tip of the manipulator arm instead of the 3D sensor, thereby performing the fastening operation using the manipulator arm. Therefore, the structure for positioning the 3D sensor at the detection location and the structure for performing the fastening operation can be made common. Consequently, it is not necessary to prepare a screw detection system for detecting the position and orientation of the male screw independently of the screw fastening system.

[0063] [Item 6] A screw detection method according to another aspect of the present disclosure is a screw detection method for detecting the position and orientation of a male screw, comprising: measuring the distance from a predetermined detection position to the male screw based on teaching data for the position and orientation of the central axis of the male screw that has been set in advance; generating a brightness distribution image of the male screw, in which the brightness differs according to the distance from the detection position to the male screw, from the distance data obtained as a result of the measurement; generating a second derivative image by taking the second derivative of the brightness distribution image; generating a third derivative image by taking the third derivative of the brightness distribution image; generating an edge extraction image showing the brightness distribution on the thread tops of the male screw as a result of calculating the logical AND of the second derivative image and the third derivative image; converting the brightness values ​​in the edge extraction image into three-dimensional point cloud data for distance; performing a cylinder fitting process to fit a cylinder model having a predetermined outer diameter from the external shape data of the male screw to the three-dimensional point cloud data; and determining the position and orientation of the central axis of the cylinder model after the execution of the cylinder fitting process as the position and orientation of the central axis of the male screw.

[0064] 1 Screw detection system 2 3D sensor 3 Processing circuit 10 Screw fastening system 12 Manipulator arm 14 Control circuit 15 Fastening jig C1 Optical axis of 3D sensor C2 Center axis of male screw G1 Brightness distribution image G2 Second derivative image G3 Third derivative image G4 Edge extraction image Mt Thread peak T Male screw U Fastened device

Claims

1. A screw detection system for detecting the position and orientation of a male screw, comprising: a three-dimensional sensor positioned at a predetermined detection position based on teaching data for the position and orientation of the central axis of the male screw, and measuring the distance to the male screw; and a processing circuit for processing the distance data from the detection position to the male screw measured by the three-dimensional sensor, wherein the processing circuit generates a brightness distribution image of the male screw, where the brightness differs depending on the distance from the detection position to the male screw, from the distance data; generates a second derivative image by taking the second derivative of the brightness distribution image; generates a third derivative image by taking the third derivative of the brightness distribution image; generates an edge extraction image showing the brightness distribution at the top of the threads of the male screw as a result of calculating the logical AND of the second derivative image and the third derivative image; converts the brightness values ​​in the edge extraction image into three-dimensional point cloud data for distance; and performs a cylinder fitting process to fit a cylinder model having a predetermined outer diameter from the outer diameter data of the male screw to the three-dimensional point cloud data. A screw detection system that determines the position and orientation of the central axis of the cylindrical model after the execution of the cylindrical fitting process as the position and orientation of the central axis of the male screw.

2. The screw detection system according to claim 1, wherein the three-dimensional sensor is positioned such that, at the detection position, its optical axis has a predetermined inclination angle with respect to the central axis of the male screw in the teaching data.

3. The screw detection system according to claim 1 or 2, comprising a manipulator arm that holds the three-dimensional sensor at its tip and whose position and orientation at the tip can be changed in three dimensions.

4. The screw detection system according to claim 1 or 2, wherein the processing circuit generates a first extracted image by extracting pixels from a plurality of pixels constituting the second derivative image whose second derivative value is between a predetermined lower limit less than 0 and 0, or between 0 and a predetermined upper limit greater than 0; generates a second extracted image by extracting pixels from a plurality of pixels constituting the third derivative image whose change in the third derivative value is within a predetermined range including 0; and generates the edge extracted image by performing a logical AND operation between the first extracted image and the second extracted image.

5. A screw fastening system comprising: a screw detection system according to claim 1 or 2; a fastening jig for holding a fixed object having a female thread corresponding to the male screw and fastening the fixed object to the male screw; a manipulator arm for selectively holding the fastening jig and the three-dimensional sensor at its tip and for changing the position and orientation of the tip in three dimensions; and a control circuit for controlling the manipulator arm, wherein the control circuit positions the three-dimensional sensor at the detection position while holding the three-dimensional sensor at the tip of the manipulator arm; compares the position and orientation of the central axis of the male screw obtained based on the distance data acquired by the three-dimensional sensor at the detection position with the position and orientation in the teaching data to correct the teaching position at the tip of the manipulator arm for fastening the fixed object to the male screw; moves the manipulator arm to the corrected teaching position while holding the fastening jig at the tip of the manipulator arm and fastens the fixed object to the male screw using the fastening jig.

6. A screw detection method for detecting the position and orientation of a male screw, comprising: measuring the distance from a predetermined detection position to the male screw based on teaching data for the position and orientation of the central axis of the male screw that has been set in advance; generating a brightness distribution image of the male screw, in which the brightness differs according to the distance from the detection position to the male screw, from the distance data obtained as a result of the measurement; generating a second derivative image by taking the second derivative of the brightness distribution image; generating a third derivative image by taking the third derivative of the brightness distribution image; generating an edge extraction image showing the brightness distribution at the top of the threads of the male screw as a result of calculating the logical AND of the second derivative image and the third derivative image; converting the brightness values ​​in the edge extraction image into three-dimensional point cloud data for distance; performing a cylinder fitting process to fit a cylinder model having a predetermined outer diameter from the external shape data of the male screw to the three-dimensional point cloud data; and determining the position and orientation of the central axis of the cylinder model after the execution of the cylinder fitting process as the position and orientation of the central axis of the male screw.

Images