Image registration method for inspecting industrial parts

Abstract

The invention relates to a method for registering a main image and a secondary image acquired by a main imaging system and by a secondary imaging system respectively, which method comprises the following steps: - acquiring (step E1) a main image of an object to be observed; - determining (step E2) an area of interest centred on a point of interest of the main image; - acquiring (step E3) a secondary image of the area of interest centred on the point of interest; - computing (step E4) a registration homography on the basis of a set of homography matrices previously obtained offline; and - registering (step E5) the secondary image with the main image by applying the computed registration homography.

Description

[0001] DESCRIPTION

[0002] TITLE: Image registration process for the inspection of industrial parts

[0003] technical field

[0004] The present invention relates to the acquisition of two images of the same object to be observed by two different imaging systems and to the registration of said two images so as to be able to combine them.

[0005] In particular, the present invention relates to the optical inspection of industrial parts, for example damaged tires, using two imaging systems with different optical properties. The registration and combination of images from these two imaging systems thus makes it possible to combine the advantages of each.

[0006] In general, the invention applies to any registration of two images, one of the two images having to image an area already present in the other image.

[0007] Previous techniques

[0008] Automating the visual analysis of manufactured objects sometimes requires combining several imaging systems with different functionalities, such as an imaging system with 3D visualization capabilities combined with another imaging system enabling thermal or multispectral observation. It is then necessary to be able to superimpose the different images acquired after registering them.

[0009] However, these solutions have drawbacks, such as excessively long visual analysis time due to excessively long calculation time or insufficiently precise control of the objects to be observed.

[0010] Furthermore, existing solutions typically involve two imaging systems fixed relative to each other and / or positioned side-by-side. This configuration allows for easy calibration and registration between the two acquired images but does not permit moving one of the imaging systems relative to the other. In addition, some registration solutions require precisely calibrating the intrinsic parameters (focal length, distortion, etc.) of the imaging systems and / or the extrinsic parameters (position, translation, rotation, etc.) of each imaging system, which is neither practical nor feasible in all cases.

[0011] Description of the invention

[0012] The present invention aims to overcome the aforementioned drawbacks and provide a method for registering images acquired by two different imaging systems of the same object to be observed, specifically a wide-field imaging system and a narrower field-of-view imaging system. Registering two images consists of determining the location of the pixels of one image in the second and thus obtaining an equivalence relationship between the two images. The present invention therefore aims to enable, on the one hand, the control of an industrial robot near the object to be observed using the wide-field imaging system, and on the other hand, the detailed analysis and study of said object with the narrower field-of-view imaging system, using the equivalence relationship between the images acquired by the two imaging systems.

[0013] The present invention relates to a method for registering a primary image and a secondary image respectively acquired by a primary imaging system fixed relative to an object to be observed and by a secondary imaging system mobile relative to said object to be observed, the field of observation of the secondary imaging system being included in the field of observation of the primary imaging system, the method comprising the following steps:

[0014] - Acquisition of a main image of the object to be observed with the main imaging system;

[0015] - Determination of an area of ​​interest centered on a point of interest in the main image;

[0016] - Acquisition of a secondary image of the area of ​​interest centered on the point of interest with the secondary imaging system; - Calculation of a registration homography from a set of homography matrices previously obtained offline, each homography matrix corresponding to a registration of an image acquired by the secondary imaging system and centered on a calibration point of an image acquired by the primary imaging system; and

[0017] - Registration of the secondary image with the main image by applying the registration homography calculated on said main image or on said secondary image.

[0018] Thus, the registration of the main image and the secondary image is carried out online but based on a set of previously obtained homography matrices, making the present process fast and reliable.

[0019] In a particular implementation mode, the set of homography matrices is obtained by the prior and offline implementation of the following steps:

[0020] - Acquisition of an image of a target comprising contrasting and random patterns by the main imaging system;

[0021] - Cutting the acquired target image into different calibration zones, each centered on a calibration point according to a calibration grid;

[0022] - Acquisition of an image of the target corresponding to each calibration zone by the secondary imaging system;

[0023] - Estimation of a homography matrix between the image acquired by the secondary imaging system and the corresponding calibration area in the image acquired by the primary imaging system; and

[0024] - Combination of each homography matrix corresponding to each calibration zone into a set of homography matrices.

[0025] Advantageously, the step of estimating a homography matrix between the image acquired by the secondary imaging system and the corresponding calibration area in the image acquired by the primary imaging system comprises the following steps:

[0026] - Resampling of images to the same pixel size; - Extraction of at least four comparison points from each image; and

[0027] - Estimation of the homography matrix from the four comparison points.

[0028] Advantageously, the step of calculating a registration homography from a set of homography matrices is implemented so that the registration homography is equal to the weighted sum of the four homography matrices corresponding to the four calibration points closest to the point of interest when the latter is superimposed on the calibration grid.

[0029] In one implementation method, the target is made in three dimensions with dimensions similar to the object to be observed, preferably by optical projection of a target onto an object similar to the object to be observed.

[0030] Advantageously, the target is positioned at a working distance from the main optical system equivalent to the distance between said main optical system and an object to be observed.

[0031] Advantageously, the primary imaging system and the secondary imaging system have a common image acquisition and / or display modality, preferably image acquisition in the same type of color register or in grayscale.

[0032] In a particular embodiment, a robot arm supports an effector or at least one of the primary imaging system or secondary imaging system, the method including a calibration step of the robot's reference frames and said primary imaging system so that the robot is controllable based on the images acquired by the primary imaging system.

[0033] Advantageously, the secondary imaging system is pre-registered approximately with respect to the primary imaging system by geometric centering of a point of an image acquired by the primary imaging system with a point of an image acquired by the secondary imaging system.

[0034] Advantageously, the object to be observed is a tire. The present invention also relates to an industrial vision system comprising a fixed main imaging system including one or two cameras adapted to acquire a three-dimensional point cloud, a mobile secondary imaging system, preferably with a multispectral or infrared sensor, whose field of observation is included in the field of observation of the main imaging system, the vision system including means for implementing the method as defined above.

[0035] The present invention also relates to a computer program comprising instructions which, when the program is executed by a computer, lead the computer to implement the steps of the process as defined above.

[0036] The system includes, for example, a computer and a computer-readable data medium, with the computer program as defined above being stored on the data medium.

[0037] Brief description of the drawings

[0038] Other objects, features and advantages of the invention will become apparent from the following description, given solely by way of non-limiting example, and made with reference to the accompanying drawings in which:

[0039] [Fig 1] is a schematic representation of the different stages of the image registration process according to the invention;

[0040] [Fig 2] is a schematic representation of an industrial vision system according to the invention;

[0041] [Fig 3] is a schematic representation of a calibration grid used during the process shown in Figure 1; and

[0042] [Fig 4] is a schematic representation of different types of positioning of points of interest on a calibration grid used during the process according to Figure 1. Detailed description of at least one embodiment

[0043] Figure 1 schematically represents the different stages of a registration process according to the invention. The process is implemented, for example, by means of an industrial vision system 1 shown in Figure 2, for example a computer 3.

[0044] The industrial vision system 1 also includes a fixed primary imaging system 5 and a mobile secondary imaging system 7. These two imaging systems 5 and 7 are suitable for imaging an object to be observed 9, for example an industrial part such as a tire.

[0045] The industrial vision system 1 optionally includes a robot 1 comprising one or more arms, one of the arms being able to support the main imaging system 5 and / or the secondary imaging system 7.

[0046] Optionally, the industrial vision system 1 includes an effector 13, in other words a tooling element, supported by a robot arm 11, allowing an intervention to be carried out on the object to be observed 9.

[0047] An intervention is, for example, a repair, and / or a manipulation, and / or an addition and / or a removal of material.

[0048] The main imaging system 5 is said to be fixed because it is adapted to image the object to be observed 9 from a fixed point at a fixed working distance Dp of said object to be observed 9, for example between 500 and 700 mm, preferably 600 mm.

[0049] The primary imaging system 5 can be a 3D imaging system comprising, for example, one or two cameras adapted to acquire a three-dimensional point cloud. Subsequently, the primary image, or any other image acquired by the primary imaging system 5, is considered to be projected onto a plane perpendicular to the direction of view, the three-dimensional point cloud being used solely for robot control.

[0050] The secondary imaging system 7 is said to be mobile because it is adapted to image the object to be observed 9 from different locations, however preferably with a fixed working distance Ds, for example between 150 and 250 mm, preferably 200 mm.

[0051] The secondary imaging system 7 can be a 2D imaging system with a multispectral or infrared sensor. Indeed, for visual inspection of pneumatics, infrared, particularly near-infrared, makes it possible to highlight defects in the wiring of metallic ribbons.

[0052] The secondary imaging system 7 includes, for example, a 25 mm focal length lens for a sensor with dimensions between 10 and 15 mm wide and between 5 and 7 mm high.

[0053] The primary imaging system 5 and the secondary imaging system 7 have a common image acquisition and / or display method; in other words, their data can be compared to detect similar features represented in the two images acquired by the two different systems. Preferably, the common method is image acquisition in the same type of color register, for example RGB, or in grayscale.

[0054] Advantageously, the field of view of the secondary imaging system 7 is smaller, for example, covering an area at least 10 times smaller, and therefore falls within the field of view of the primary imaging system 5. For example, the dimensions of the field of view of the secondary imaging system 7 are between 75 and 85 mm wide and between 35 and 50 mm high. Thus, the primary imaging system 5 provides an overview of the object to be observed 9, for example, to guide the operator in space for an intervention on said object 9. The secondary imaging system 7 allows specific information to be obtained on certain areas, for example, damaged areas, of the object 9.

[0055] Prior to implementing steps E1 to E5 for image registration, the method optionally includes a PI step for calibrating the reference frames of robot 11 and the main imaging system 5 so that robot 11 can be controlled based on the images acquired by the main imaging system 5. This PI step consists, for example, of mounting a high-contrast checkerboard at the end of robot 11's arm, then positioning the checkerboard in different locations within the field of view of the main imaging system 5 and taking an image in each of these positions. All of these images are then used to estimate the position of the main imaging system 5 in robot 11's reference frame and thus obtain a calibration of their respective reference frames.

[0056] For the implementation of the image registration process illustrated in Figure 1, a first step E is carried out of acquiring a main image of the object to be observed 9 with the main imaging system 5.

[0057] This main image is therefore taken with a field of view allowing a wide view of the object to be observed 9.

[0058] Then, we perform step E2 to determine an area of ​​interest centered on an interest point of the main image.

[0059] This step E2 is for example carried out manually by an operator pointing to a point of interest, for example towards an area of ​​interest where there appears to be damage to the object to be observed 9.

[0060] Alternatively, this step E2 is performed automatically by a computer, for example equipped with a neural network learning module. The computer detects damage and determines a zone of interest centered on a point of interest at that location.

[0061] Then, we perform an E3 step of acquiring a secondary image of the area of ​​interest centered on the point of interest with the secondary imaging system 7.

[0062] This secondary image will be a view whose field of observation will be reduced compared to the main image, but allows to highlight other characteristics, in particular thanks to the possible infrared properties of the sensor of the secondary imaging system 7.

[0063] Clearly, for the implementation of this step E3, the secondary imaging system 7 is first registered, at least approximately, with respect to the primary imaging system 5 so that it can image substantially the correct area of ​​interest. For example, a registration step P2 of the secondary imaging system 7 with respect to the primary imaging system 5 is performed by geometrically centering a point in an image acquired by the primary imaging system 5 with a point in an image acquired by the secondary imaging system 7.

[0064] Then, we perform a step E4 of calculating a registration homography from a set of homography matrices previously obtained offline, each homography matrix corresponding to a registration of an image acquired by the secondary imaging system 7 and centered on a calibration point of an image acquired by the main imaging system 5.

[0065] Finally, we perform a step E5 of registration of the secondary image with the main image by applying the registration homography calculated on said main image or on said secondary image.

[0066] In general, steps E1, E2, E3, E4, and E5 are designed for online implementation, that is, within an industrial control or production chain. These steps E1 through E5 have the advantage of utilizing a set of homography matrices previously obtained offline through lengthy and tedious steps H1 through H5, detailed below. "Offline" refers to the steps H1 through H5 implemented in preparation for steps E1 through E5, and more specifically for the acquisition and registration of the primary and secondary images.

[0067] Unlike the prior art, the present invention is therefore a rapid registration method, since some solutions of the prior art also consist of using homography matrices, but whose estimation is carried out online and is therefore costly in computation time.

[0068] The set of homography matrices is obtained by prior and offline implementation of the following steps H1 to H5.

[0069] Firstly, an H1 step is performed to acquire an image of a target comprising contrasting and random patterns by the main imaging system 5. The target can be in two dimensions but is preferably made in three dimensions with dimensions similar to those of the object to be observed 9. In the case of a tire, the target is for example a cylinder or a portion of a cylinder with a radius of curvature equivalent to that of a tire.

[0070] The target can be obtained by three-dimensional printing or by molding said target.

[0071] Alternatively, the target is obtained by optical projection of said target onto an object similar to the object to be observed 9. Advantageously, a projector illuminates a defect-free tire with an image of a target.

[0072] Advantageously, the contrast of the target can be achieved by a mixture of light and dark colors, such as black and white.

[0073] The random patterns of the target are obtained, for example, by black spots of varying sizes on a white background, or vice versa. This target makes the present method robust and applicable to any object to be observed, whether textured or not.

[0074] The target is preferably positioned at a working distance Dp from the main optical system equivalent to the distance Dp between said main optical system and the object to be observed 9 in line.

[0075] Then, we perform an H2 step of cutting the image of the acquired target into different calibration zones, each centered on a calibration point according to a calibration grid.

[0076] The calibration grid is, in other words, a set of calibration points forming calibration zones centered on each of these calibration points.

[0077] The division of step H2 is carried out for example into N rows and C columns forming a predefined number of calibration zones, for example a number greater than 50.

[0078] Alternatively, each calibration point of the calibration grid is spaced from another nearest calibration point by a distance equivalent to half the angular deviation defined by the field of view of the secondary imaging system 7. Thus, a calibration area is, for example, the size of an image acquired by the secondary imaging system 7.

[0079] Then, an H3 step is performed to acquire an image of the target corresponding to each calibration zone by the secondary imaging system 7. In practice, this H3 step is carried out for the N x C calibration zones.

[0080] Next, we perform an H4 step estimating a homography matrix between each image acquired by the secondary imaging system 7 and each corresponding calibration area in the image acquired by the main imaging system 5.

[0081] In image processing, homography generally describes the transformation that occurs when two identical plane surfaces are projected onto two different images; the two plane surfaces are thus viewed from two different angles. In our case, the area observed, or calibration area, by the secondary imaging system 7 will not necessarily be planar, but this assumption is acceptable because the field of view of the secondary imaging system 7 is very small.

[0082] Step H4 of estimating a homography matrix between the image acquired by the secondary imaging system 7 and the corresponding calibration area in the image acquired by the main imaging system 5 includes, for example, the following substeps H41, H42 and H43.

[0083] First, an H41 step is performed to resample the images to the same pixel size. Since both images have the same size, it will then be possible to estimate a homography matrix linking them through a transformation.

[0084] Next, an H42 step is performed to extract at least four comparison points from each image. The comparison points are similar in pairs and therefore represent the same location on the target, but imaged on two different images. The purpose of the H4 step is to estimate what transformation occurs from one image to the other that could change the pixels of these comparison points. This H42 step is, for example, performed using a method known in the state of the art, namely a Harris, SIFT, or SURF type method. The common image acquisition and / or display method between the primary imaging system 5 and the secondary imaging system 7 is particularly advantageous here since it allows the method to easily detect the similarities between the two images.

[0085] Finally, we perform an H43 step of estimating the homography matrix from the four comparison points, for example using a known state-of-the-art method such as a RANSAC type method.

[0086] Formally, we can define (x, y) a pixel of a calibration area imaged by the main imaging system 5.

[0087] Similarly, we define (x', y') as the pixel of the image acquired by the secondary imaging system 7.

[0088] With s being the scale factor related to the projection and H the homography matrix between these two pixels and by extension between the two images, we obtain: . . . J

[0089] In practice, to fix the scale factor s, we normalize h33 to be equal to 1 and we therefore have 8 terms left in the 3x3 matrix to determine, which represents 8 degrees of freedom determinable from the four pairs of comparison points.

[0090] Finally, we perform an H5 step combining each homography matrix corresponding to each calibration zone into a set of homography matrices.

[0091] The set of homography matrices can thus be stored by the computer and used during step E4 to determine a specific online registration homography from the previously obtained homography matrices.

[0092] In particular, step E4 of calculating a registration homography from the set of homography matrices is implemented so that the registration homography is equal to the weighted sum of the four homography matrices corresponding to the four calibration points closest to the point of interest when the latter is superimposed on the calibration grid.

[0093] A calibration grid is shown in Figure 3 superimposed with a main image of a tire.

[0094] In particular, for a point of interest, symbolized by a cross, positioned between 4 calibration points located respectively at a distance d1, d2, d3 and d4 from the point of interest and respectively at the 4 eme line 2 eme column (homography matrix denoted H42), and 4 eme line 3 eme column (homography matrix denoted H43), and 5 eme line 2 eme column (homography matrix denoted H52), and 5 eme line 3 eme column

[0095] (homography matrix denoted H53) of the calibration grid, the registration homography H' is calculated such that:

[0096] 1

[0097] H' = — - - - - - (wl x H42 + w2 x H43 + w3 x H52 + w4 x H53) wl + w2 + w3 + w4 7

[0098] 1 1 1 1

[0099] With wl = - , w2 = - , w3 = - , w4 = - , dl / D d2 / D d3 / D d4 / D

[0100] And with D = dl + d.2 + d3 + d4.

[0101] This calculation works for a point of interest located in the middle of several calibration points of the calibration grid.

[0102] Figure 4 illustrates in particular three different cases of figures, with the first case of figures having cross-shaped points representing points of interest located in the middle of four calibration points, represented as circles, of the calibration grid, as described previously.

[0103] In the second scenario, the calculation can be modified to take into account only the calibration point closest to the point of interest, represented as a triangular point, when said point of interest is located on a corner of the main image, or on an edge of the main image and aligned on the same row or column as calibration points of the calibration grid.

[0104] In a third scenario, the calculation can also be modified to consider only the two calibration points closest to the point of interest, represented as a square point, when said point of interest is located on an edge of the main image and not aligned in the same row or column as calibration points of the calibration grid, or when said point of interest is not located on an edge of the main image and is located in the same row or column as calibration points of the calibration grid. Once completed, this process can then be used to perform an intervention step on the object being observed. For example, in the case of a tire, a step involving degreasing, machining, polishing, welding, or retreading can be implemented.

Claims

DEMANDS 1. A method for registering a primary image and a secondary image respectively acquired by a primary imaging system (5) fixed relative to an object to be observed (9) and by a secondary imaging system (7) mobile relative to said object to be observed (9), the field of observation of the secondary imaging system (7) being included in the field of observation of the primary imaging system (5), characterized in that it comprises the following steps: - Acquisition (step E l) of a main image of the object to be observed (9) with the main imaging system (5); Determination (step E2) of an area of ​​interest centered on a point of interest in the main image; - Acquisition (step E3) of a secondary image of the area of ​​interest centered on the point of interest with the secondary imaging system (7); Calculation (step E4) of a registration homography from a set of homography matrices previously obtained offline, each homography matrix corresponding to a registration of an image acquired by the secondary imaging system (7) and centered on a calibration point of an image acquired by the primary imaging system (5); and Registration (step E5) of the secondary image with the main image by applying the registration homography calculated on said main image or on said secondary image.

2. A method according to claim 1, wherein the set of homography matrices is obtained by prior and offline implementation of the following steps: Acquisition (step H l) of an image of a target comprising contrasting and random patterns by the main imaging system (5); Segmentation (step H2) of the acquired target image into different calibration zones, each centered on a calibration point according to a calibration grid; - Acquisition (step H3) of an image of the target corresponding to each calibration zone by the secondary imaging system (7); Estimation (step H4) of a homography matrix between the image acquired by the secondary imaging system (7) and the corresponding calibration area in the image acquired by the primary imaging system (5); and Combination (step H5) of each homography matrix corresponding to each calibration zone into a set of homography matrices.

3. A method according to claim 2, wherein the step (H4) of estimating a homography matrix between the image acquired by the secondary imaging system (7) and the corresponding calibration area in the image acquired by the primary imaging system (5) comprises the following steps: Resampling (step H41) of the images to the same size in number of pixels; Extraction (step H42) of at least four comparison points in each image; and Estimation (step H43) of the homography matrix from the four comparison points.

4. A method according to any one of claims 2 and 3, wherein the step (E4) of calculating a registration homography from a set of homography matrices is implemented such that the registration homography is equal to the weighted sum of the four homography matrices corresponding to the four calibration points closest to the point of interest when the latter is superimposed on the calibration grid.

5. A method according to any one of claims 2 to 4, wherein the target is made in three dimensions according to dimensions similar to the object to be observed (9), preferably by optical projection of a target onto an object similar to the object to be observed (9).

6. Method according to any one of claims 2 to 5, wherein the target is positioned at a working distance from the main optical system equivalent to the distance between said main optical system and an object to be observed (9).

7. A method according to any one of claims 1 to 6, wherein the main imaging system (5) and the secondary imaging system (7) have a common image acquisition and / or display mode, preferably image acquisition in the same type of color register or in grayscale.

8. A method according to any one of claims 1 to 7, wherein a robot arm (11) supports an effector (13) or at least one of the primary imaging system (5) or of the secondary imaging system (7), the method comprising a step (PI) of calibrating the reference frames of the robot (11) and said primary imaging system (5) so that the robot (11) is controllable based on the images acquired by the primary imaging system (5).

9. A method according to any one of claims 1 to 8, wherein the secondary imaging system (7) is first registered (step P2) approximately with respect to the primary imaging system (5) by geometric centering of a point of an image acquired by the primary imaging system (5) with a point of an image acquired by the secondary imaging system (7).

10. Method according to any one of claims 1 to 9, wherein the object to be observed (9) is a tire. 1 1. Industrial vision system (1) comprising a fixed primary imaging system (5) including one or two cameras adapted to acquire a three-dimensional point cloud, a mobile secondary imaging system (7), preferably with a multispectral or infrared sensor, the field of view of which is included in the field of view of the primary imaging system (5), the vision system comprising means (3) for implementing the method according to any one of claims 1 to 10.

Images