Image distortion correction method and robot system

Abstract

This image distortion correction method involves imaging a workpiece from a prescribed height using a two-dimensional camera, and correcting the distortion in the captured image using a lens distortion coefficient of the two-dimensional camera. A first step involves measuring the lens distortion coefficient of the two-dimensional camera at a plurality of different working distances. A second step involves obtaining the relationship between the working distances and the lens distortion coefficients in the two-dimensional camera from the measurement result of the first step. A third step involves measuring the working distance between the two-dimensional camera and the workpiece at the installation height of the two-dimensional camera when the two-dimensional camera is installed, and obtaining the lens distortion coefficient at the installation height of the two-dimensional camera on the basis of the measured working distance and the relationship obtained in the second step.

Description

Image distortion correction method and robot system

[0001] The present specification discloses an image distortion correction method and a robot system.

[0002] Conventionally, techniques for correcting distortion caused by a lens in an image captured by a camera are known. For example, Patent Document 1 discloses the image distortion correction method described below. First, during product assembly in a factory, test images are captured at a plurality of different focal lengths of an optical lens system, and for each focal length of the optical lens system, coordinates on the test image and coordinates on an output image of an imaging device are associated with each other and recorded in advance in a table. Next, during operation of the imaging device, the focal length of the optical lens system and the readout position of the image sensor are acquired, and coordinates on the test image corresponding thereto are read from the table as distortion-corrected coordinates. Then, the captured image is corrected by replacing the coordinates of the image signal distorted under the influence of distortion aberration with the distortion-corrected coordinates.

[0003] Japanese Unexamined Patent Publication No. 2007-129587

[0004] The distortion characteristic of a lens (lens distortion coefficient) also changes depending on the working distance, which is the distance between the lens and a workpiece. For this reason, depending on the installation conditions of the camera or the workpiece, distortion of a captured image may not be corrected correctly. For example, when correcting distortion in an image of a workpiece captured by a fixed camera installed on a ceiling or the like, a wide field of view is often required, and it is difficult to prepare a calibration plate (test image) of a size matching the required field of view. Further, since measurement using the calibration plate must be performed at the site where the camera is installed, it is also necessary to dispatch specialized workers to the site. Furthermore, when the installation conditions of the workpiece change, or when a camera is attached to a robot arm for imaging, if the working distance deviates from the reference value, accurate distortion correction cannot be performed, so it is necessary to restart the process from measurement using the calibration plate.

[0005] A main object of the present disclosure is to simply correct distortion caused by a lens in a captured image with favorable accuracy, regardless of the installation conditions of a camera or a workpiece.

[0006] This disclosure employs the following means to achieve the primary objectives described above.

[0007] The present disclosure provides an image distortion correction method that captures an image of a workpiece from a predetermined height using a two-dimensional camera and corrects the distortion of the captured image using the lens distortion coefficient of the two-dimensional camera, comprising: a first step of measuring the lens distortion coefficient of the two-dimensional camera at a plurality of different working distances; a second step of determining the relationship between the working distance and the lens distortion coefficient of the two-dimensional camera from the measurement results of the first step; and a third step of measuring the working distance between the two-dimensional camera and the workpiece at the installation height of the two-dimensional camera when the two-dimensional camera is installed, and determining the lens distortion coefficient at the installation height of the two-dimensional camera based on the measured working distance and the relationship determined in the second step.

[0008] In this disclosed image distortion correction method, the lens distortion coefficient of a two-dimensional camera is measured at multiple different working distances, and the relationship between the working distance and the lens distortion coefficient in the two-dimensional camera is determined in advance from the measurement results. As a result, the lens distortion coefficient at a working distance can be determined with good accuracy simply by measuring the working distance between the two-dimensional camera and the workpiece. Consequently, distortion of the captured image caused by the lens can be corrected simply and with good accuracy, regardless of the installation conditions of the camera and workpiece.

[0009] The robot system of this disclosure comprises a work robot that performs work on a workpiece and a two-dimensional camera that images the workpiece from a predetermined height, and the system comprises a storage unit that stores the relationship between the working distance and the lens distortion coefficient of the two-dimensional camera, a derivation unit that acquires the working distance between the two-dimensional camera and the workpiece at the installation height of the two-dimensional camera and derives the lens distortion coefficient at the installation height of the two-dimensional camera based on the acquired working distance and the relationship stored in the storage unit, and an image processing unit that corrects the distortion of the image of the workpiece captured by the two-dimensional camera using the lens distortion coefficient derived by the derivation unit and recognizes the position of the workpiece.

[0010] In the robot system of this disclosure, the relationship between the working distance and the lens distortion coefficient of the two-dimensional camera is stored in a memory unit beforehand. This allows the lens distortion coefficient at the working distance to be derived with good accuracy simply by acquiring the working distance between the two-dimensional camera and the workpiece. As a result, distortion of the captured image caused by the lens can be corrected easily and with good accuracy, regardless of the camera and workpiece installation conditions. Furthermore, the position of the workpiece can be accurately recognized from the distortion-corrected captured image.

[0011] This is a schematic diagram of a robot system. This is an explanatory diagram of a system in which a two-dimensional camera is mounted on a robot arm. This is an explanatory diagram of a system in which a two-dimensional camera is fixed above the work area. This is a block diagram showing the electrical connection relationships of the robot system. This is an explanatory diagram showing how the position of a workpiece is recognized relative to an image. This is a flowchart showing an example of an image distortion correction procedure. This is a flowchart showing an example of preparation before camera adjustment. This is an explanatory diagram showing an example of a calibration plate. This is an explanatory diagram showing an example of the relationship between the working distance WD and the first-order distortion coefficient K1. This is an explanatory diagram showing an example of the relationship between the working distance WD and the second-order distortion coefficient K2. This is an explanatory diagram showing an example of the relationship between the working distance WD and the third-order distortion coefficient K3. This is an explanatory diagram showing an example of the relationship between the reciprocal of the square of the working distance WD and the first-order distortion coefficient K1. This is an explanatory diagram showing an example of the relationship between the reciprocal of the fourth-order working distance WD and the second-order distortion coefficient K2. This is an explanatory diagram showing an example of the relationship between the reciprocal of the sixth-order working distance WD and the third-order distortion coefficient K3. This is a flowchart showing an example of preparation before camera installation. This is an explanatory diagram showing the derivation of the first-order distortion coefficient K1_n at the working distance WDn after camera installation. It is a flowchart illustrating an example of image distortion correction.

[0012] Next, the forms for implementing this disclosure will be described with reference to the drawings.

[0013] Figure 1 is a schematic diagram of the robot system 10. Figure 2A is an explanatory diagram of a system in which a two-dimensional camera 30 is installed on the robot arm 22. Figure 2B is an explanatory diagram of a system in which the two-dimensional camera 30 is fixed above the work area. Figure 3 is a block diagram showing the electrical connection relationships of the robot system 10.

[0014] As shown in Figure 1, the robot system 10 comprises a workpiece supply device 12, a robot 20, a two-dimensional camera 30, and a control device 40 (see Figure 3). The robot system 10 uses the two-dimensional camera 30 to image the workpiece W (object to be worked on) supplied by the workpiece supply device 12 to recognize the position of the workpiece W, and the robot 20 performs work on the workpiece W at the recognized position. Examples of tasks performed by the robot 20 include transporting the workpiece W and moving it to another location, and assembly, where the workpiece W is picked up and assembled onto an object to be assembled.

[0015] In this embodiment, the robot 20 includes a five-axis vertical articulated arm (hereinafter referred to as the arm) 22, as shown in Figures 1 and 2. The arm 22 has six links and five joints that connect each link so that it can rotate or pivot. Each joint is provided with a motor (servo motor) 24 that drives the corresponding joint and an encoder (rotary encoder) 26 that detects the rotational position of the corresponding motor 24.

[0016] A work tool, which functions as an end effector, is detachably attached to the tip link of arm 22. Examples of work tools include electromagnetic chucks, mechanical chucks, and suction nozzles.

[0017] The two-dimensional camera 30 is positioned to face vertically downwards and captures images of the workpiece W placed on the workpiece mounting surface S in the work area. The two-dimensional camera 30 can be mounted on the arm 22 (see Figure 2A), or on a fixing member M fixed above the work area of ​​the robot 20 (for example, the ceiling portion of the protective fence of the robot 20) (see Figure 2B). The distance between the lens of the two-dimensional camera 30 and the workpiece W (working distance WD) varies depending on the height of the mounting surface of the two-dimensional camera 30, the height of the mounting surface of the workpiece W, the height of the workpiece W, etc.

[0018] The robot control device 40 is configured as a microprocessor centered around a CPU. The robot control device 40 includes, as functional blocks, a robot control unit 41 that controls each of the robot's motors 24, an image processing unit 42 that processes images captured by the two-dimensional camera 30, and a storage unit 43 that stores various data. The robot control device 40 receives input such as position signals from each encoder 26 and image signals from the two-dimensional camera 30. The robot control device 40 also outputs control signals to each of the motors 24 and control signals to the two-dimensional camera 30.

[0019] The robot control unit 41 moves the work tool attached to the end link of the arm 22 toward the workpiece W and controls each motor 24 and the work tool to perform work on the workpiece W using the work tool. Specifically, the robot control device 40 captures an image of the workpiece W with the two-dimensional camera 30 and obtains the position coordinates (Xwr, Ywr) of the workpiece W in the robot coordinate system from the image processing unit 42. Next, the robot control unit 41 sets the target position (Xt, Yt, Zt) and target orientation (RB, RC) of the work tool based on the acquired position coordinates (Xwr, Ywr) of the workpiece W, and transforms the set target position (Xt, Yt, Zt) and target orientation (RB, RC) into target angles for each joint of the arm 22 using well-known HD parameters, etc. Then, the robot control unit 41 controls the corresponding motor 24 so that the angle of each joint detected by each encoder 26 matches the respective target angle, and controls the work tool so that work is performed on the workpiece W.

[0020] The image processing unit 42 processes the image of the workpiece W captured by the two-dimensional camera 30 to recognize the position coordinates (Xwr, Ywr) of the workpiece W in the robot coordinate system. Specifically, the image processing unit 42 first performs image distortion correction on the image of the workpiece W to correct for distortion caused by the lens of the two-dimensional camera 30. Image distortion correction is performed by reading the distance between the lens of the two-dimensional camera 30 and the workpiece W (working distance WDn), which is stored in advance in the memory unit 43, and using the read working distance WDn, converting the coordinates (X, Y) of each pixel in the image of the workpiece W to corrected coordinates (X_corrected, Y_corrected) using equations (1) and (2). Here, in equations (1) and (2), "K1_n" is the first-order radial distortion coefficient (first-order distortion coefficient) at the working distance WDn, "K2_n" is the second-order radial distortion coefficient (second-order distortion coefficient), and "K3_n" is the third-order radial distortion coefficient (third-order distortion coefficient). Also, "r" is the distance from the coordinates of the central pixel to the coordinates (X, Y) of the pixel to be corrected, which includes the distortion.

[0021]

[0022] Next, the image processing unit 42 recognizes the pixel-level position coordinates (Xwv, Ywv) [pixels] of the workpiece W in a Cartesian coordinate system (image coordinate system) with the origin O being the reference point of the image (for example, the center of the image) (see Figure 4). Subsequently, the image processing unit 42 converts the pixel-level position coordinates (Xwv, Ywv) [pixels] of the workpiece W into length-level position coordinates (Xwv, Ywv) [μm] using equation (3). In equation (3), "Res" is the size (resolution) of one pixel on the upper surface of the workpiece W, and is stored in advance in the storage unit 43. Then, the image processing unit 42 converts the length-level position coordinates (Xwv, Ywv) [μm] of the workpiece W in the image coordinate system and the workpiece height h into three-dimensional robot coordinate coordinates (Xwr, Ywr, Zwr).

[0023] (Xw,Yw)[pixel]=(Xw,Yw)[μm]×Res…(3)

[0024] Next, the procedure for image distortion correction will be explained. Figure 5 is a flowchart showing an example of the image distortion correction procedure. Image distortion correction is performed by sequentially executing the following: pre-preparation during camera adjustment (step S100), pre-preparation during camera installation (step S110), and image distortion correction during production (step S120).

[0025] Figure 6 is a flowchart showing an example of pre-adjustment for camera adjustment. Pre-adjustment for camera adjustment is performed by the image processing unit 42 before the robot system 10 is shipped to the customer.

[0026] In the preliminary preparations for camera adjustment, the image processing unit 42 first obtains measured values ​​of the lens distortion coefficients K1_s, K2_s, and K3_s at a reference working distance WDs (step S200). Subsequently, the image processing unit 42 obtains measured values ​​of the respective lens distortion coefficients K1_i, K2_i, and K3_i (i=1,2,3) at a plurality of working distances WDi (i=1,2,3) that are different from the reference working distance WDs (step S210). For example, the working distances WDs+d, WDs+2×d, and WDs+3×d are defined, each located at a predetermined distance d (for example, 20 mm) from the reference working distance WDs.

[0027] The lens distortion coefficients K1_s, K2_s, K3_s, K1_i, K2_i, K3_i (i=1,2,3) are measured as follows. First, a calibration plate (see Figure 7) is set up, for example, in which circular dots are arranged in a matrix at equal intervals L. Next, the distance between the lens of the two-dimensional camera 30 and the calibration plate CP is adjusted to the respective working distances WDs, WDi (i=1,2,3), and the calibration plate CP is imaged by the two-dimensional camera 30. Then, the position of the dots is recognized from each captured image, and the measured values ​​of the lens distortion coefficients K1_s, K2_s, K3_s, K1_i, K2_i, K3_i (i=1,2,3) are calculated by applying a well-known distortion coefficient calculation program to the recognized dot positions.

[0028] Next, the image processing unit 42 derives unit distortion coefficients K1_unit, K2_unit, and K3_unit from the measured values ​​of lens distortion coefficients K1_s, K2_s, K3_s, K1_i, K2_i, and K3_i (i=1,2,3) at four working distances WDs and WDi (i=1,2,3) (step S220). Here, the first-order unit distortion coefficient K1_unit is the slope of the linear function between the reciprocal of the square of the working distance WD and the first-order distortion coefficient K1. The second-order unit distortion coefficient K2_unit is the slope of the linear function between the reciprocal of the fourth-order working distance WD and the second-order distortion coefficient K2. The third-order unit distortion coefficient K3_unit is the slope of the linear function between the reciprocal of the sixth-order working distance WD and the third-order distortion coefficient K3. In this embodiment, the first-order unit strain coefficient K1_unit is derived by calculating the slope of the regression line using the least squares method for combinations of multiple working distances WDs, WDi (i = 1, 2, 3) and their corresponding first-order strain coefficients K1_s, K1_i (i = 1, 2, 3). The second-order unit strain coefficient K2_unit is derived by calculating the slope of the regression line using the least squares method for combinations of multiple working distances WDs, WDi (i = 1, 2, 3) and their corresponding second-order strain coefficients K2_s, K2_i (i = 1, 2, 3). The derivation of the third-order unit strain coefficient K3_unit is performed by calculating the slope of the regression line using the least squares method for combinations of multiple working distances WDs, WDi (i = 1, 2, 3) and their corresponding third-order strain coefficients K3_s, K3_i (i = 1, 2, 3). As shown in Figures 8A, 8B, and 8C, the first-order strain coefficient K1, the second-order strain coefficient K2, and the third-order strain coefficient K3 are each related to the working distance WD by a curve. In contrast, as a result of diligent research, the inventors of the present invention have found that, as shown in Figures 9A, 9B, and 9C, the first-order strain coefficient K1, the second-order strain coefficient K2, and the third-order strain coefficient K3 can be well approximated by a linear function with respect to the reciprocal of the square of the working distance WD, the reciprocal of the fourth power of the working distance WD, and the reciprocal of the sixth power of the working distance WD, respectively.Therefore, by deriving the unit distortion coefficients K1_unit, K2_unit, and K3_unit in advance, the lens distortion coefficients K1, K2, and K3 can be derived with good accuracy from the unit distortion coefficients K1_unit, K2_unit, and K3_unit for any working distance WD.

[0029] In this embodiment, regression analysis is performed using four working distances WDs and WDi (i = 1, 2, 3). However, regression analysis may also be performed using measurements at more different working distances WD, or using multiple measurements at the same working distance WD. Doing so can be expected to improve the accuracy of the regression analysis.

[0030] Having derived the unit distortion coefficients K1_unit, K2_unit, and K3_unit in this way, the image processing unit 42 stores the reference working distance WDs, the lens distortion coefficients K1_s, K2_s, and K3_s at the reference working distance WDs, and the unit distortion coefficients K1_unit, K2_unit, and K3_unit in the storage unit 43 (step S230), thereby completing the preliminary preparations for camera adjustment.

[0031] Next, we will explain the preparations made before camera installation. Figure 10 is a flowchart showing an example of the preparations made before camera installation. The preparations for camera installation are performed by the image processing unit 42 when the two-dimensional camera 30 is installed at the destination (production facility) after the robot system 10 has been shipped to the customer.

[0032] In the preliminary preparations for camera installation, the image processing unit 42 first acquires the installation height (working distance WDn) of the two-dimensional camera 30 relative to the installation surface S of the workpiece W (step S300). In step S300, the image processing unit 42 may acquire an actual measured value of the working distance WDn, or it may acquire an estimated value of the working distance WDn by estimating it from the image of the mounting surface S captured by the two-dimensional camera 30 and the internal parameters of the two-dimensional camera 30 (such as the size of the image sensor and the focal length). Next, the image processing unit 42 calculates the lens distortion coefficients K1_n, K2_n, and K3_n at the working distance WDn using equations (4), (5), and (6) based on the working distance WDn, the reference working distance WDs stored in the memory unit 43 during the preparation for camera adjustment, the lens distortion coefficients K1_s, K2_s, and K3_s at the reference working distance WDs, and the unit distortion coefficients K1_unit, K2_unit, and K3_unit (step S310). Figure 11 shows how the first-order distortion coefficient K1_n at the working distance WDn after camera installation is derived. Equation (4) can be easily derived from Figure 11. The same applies to equations (5) and (6).

[0033]

[0034] Then, the image processing unit 42 stores the lens distortion coefficients K1_n, K2_n, and K3_n at the working distance WDn in the storage unit 43 (step S320), and completes the preliminary preparations for camera installation.

[0035] Next, we will explain image distortion correction. Figure 12 is a flowchart showing an example of image distortion correction. Image distortion correction is performed by the image processing unit 42 during production.

[0036] In image distortion correction, the image processing unit 42 first acquires an image of the workpiece W captured by the two-dimensional camera 30 during production (step S400). Next, the image processing unit 52 reads the lens distortion coefficients K1_n, K2_n, and K3_n at the working distance WDn from the storage unit 43 (step S410). Then, the image processing unit 42 performs distortion correction on the image of the workpiece W using the above-described equations (1) and (2) (step S420). Finally, the image processing unit 42 recognizes the position coordinates (Xw, Yw) of the workpiece W by performing image processing on the distortion-corrected image (step S430), and completes the image distortion correction. As a result, even if the working distance WDn during production changes significantly from the reference working distance WD0 due to changes in the installation height of the two-dimensional camera 30 or the workpiece W, simply by acquiring the working distance WDn, the lens distortion coefficients K1_n, K2_n, and K3_n of the two-dimensional camera 30 can be derived with good accuracy, and the position of the workpiece W can be recognized with high accuracy. As a result, by controlling the robot 20 based on the recognized position of the workpiece W, the accuracy of the work performed by the robot 20 on the workpiece W can be further improved.

[0037] In this embodiment, the image processing unit 42 stores the relationship between the reciprocal of an even power of the working distance WD and the lens distortion coefficients K1, K2, K3 (see Figures 9A, 9B, and 9C) in the storage unit 43 using a linear approximation formula (unit distortion coefficients K1_unit, K2_unit, K3_unit), and derives the corresponding lens distortion coefficients K1_n, K2_n, and K3_n from the stored approximation formula for any working distance WDn. However, the relationship between the working distance WD and the lens distortion coefficients K1, K2, K3 (see Figures 8A, 8B, and 8C) may be stored in the storage unit 43 using a higher-order approximation formula or map, and the corresponding lens distortion coefficients K1_n, K2_n, and K3_n may be derived from the stored approximation formula or map for any working distance WDn.

[0038] In this embodiment, the robot 20 is equipped with a five-axis vertical articulated arm 22. However, the number of axes of the arm is not limited to five; it may be four or fewer axes, or six or more axes. The robot may also be equipped with a horizontal articulated arm.

[0039] In this embodiment, the image distortion correction method of the present disclosure has been described in relation to a robot system 10, but it may be applied to any device or system that recognizes the position of a workpiece through image processing.

[0040] As described above, the image distortion correction method of this disclosure measures the lens distortion coefficient of a two-dimensional camera at multiple different working distances, and pre-determines the relationship between the working distance and the lens distortion coefficient of the two-dimensional camera from the measurement results. This makes it possible to determine the lens distortion coefficient at a working distance with good accuracy simply by measuring the working distance between the two-dimensional camera and the workpiece. As a result, distortion of the captured image caused by the lens can be corrected simply and with good accuracy, regardless of the installation conditions of the camera and workpiece.

[0041] In the image distortion correction method of the present disclosure, the second step may determine a unit distortion coefficient which is related to the reciprocal of an even power of the working distance and the lens distortion coefficient, and the third step may determine the lens distortion coefficient at the installation height of the two-dimensional camera based on the working distance at the installation height of the two-dimensional camera and the unit distortion coefficient determined in the second step. This is based on the discovery by the inventors of the present application that there is a constant relationship between the reciprocal of an even power of the working distance and the distortion coefficient of the corresponding order. In this case, the second step may determine a linear function relationship as the unit distortion coefficient. In this way, the range distortion coefficient can be calculated with good accuracy from any working distance by simple calculation.

[0042] This disclosure is not limited to the form of an image distortion correction method, but can also be in the form of a robotic system.

[0043] It should be noted that the present invention is not limited to the above-described embodiments in any way, and it goes without saying that the present invention can be implemented in various aspects as long as it falls within the technical scope of the present disclosure.

[0044] The present disclosure is applicable to manufacturing industries such as robot systems and image processing apparatuses.

[0045] 10 Robot system, 12 Work supply device, 20 Robot, 22 Arm, 24 Motor, 26 Encoder, 30 Two-dimensional camera, 31 Lens, 40 Robot control device, 41 Robot control unit, 42 Image processing unit, 43 Storage unit, CP Calibration plate, M Fixing member, S Mounting surface.

Claims

1. An image distortion correction method for imaging a workpiece from a predetermined height with a two-dimensional camera and correcting the distortion of the image using the lens distortion coefficient of the two-dimensional camera, comprising: a first step of measuring the lens distortion coefficient of the two-dimensional camera at a plurality of different working distances; a second step of determining the relationship between the working distance and the lens distortion coefficient of the two-dimensional camera from the measurement results of the first step; and a third step of measuring the working distance between the two-dimensional camera and the workpiece at the installation height of the two-dimensional camera when the two-dimensional camera is installed, and determining the lens distortion coefficient at the installation height of the two-dimensional camera based on the measured working distance and the relationship determined in the second step.

2. An image distortion correction method according to claim 1, wherein the second step is to determine a unit distortion coefficient which is related to the reciprocal of an even power of the working distance and the lens distortion coefficient, and the third step is to determine the lens distortion coefficient at the installation height of the two-dimensional camera based on the working distance at the installation height of the two-dimensional camera and the unit distortion coefficient determined in the second step.

3. An image distortion correction method according to claim 2, wherein the second step is to determine a linear relationship as the unit distortion coefficient.

4. A robot system comprising a work robot that performs work on a workpiece, and a two-dimensional camera that images the workpiece from a predetermined height, the robot system comprising: a storage unit that stores the relationship between the working distance and the lens distortion coefficient of the two-dimensional camera; a derivation unit that acquires the working distance between the two-dimensional camera and the workpiece at the installation height of the two-dimensional camera, and derives the lens distortion coefficient at the installation height of the two-dimensional camera based on the acquired working distance and the relationship stored in the storage unit; and an image processing unit that corrects the distortion of the image of the workpiece captured by the two-dimensional camera using the lens distortion coefficient derived by the derivation unit and recognizes the position of the workpiece.

Images