Multi-camera 3D calibration device and method based on multi-z-axis moving grid patterns

Abstract

It was necessary to use two types of jigs and perform a separate process for each jig in the past in order to precisely perform multi-camera three-dimensional (3D) calibration. In order to solve this, the technical idea of the present disclosure makes it possible to perform an image calibration process at a time and maintain precision by means of multi-camera 3D calibration device and method based on multi-Z-axis moving grid patterns. In addition, the multi-camera 3D calibration device and method based on multi-Z-axis moving grid patterns according to an embodiment provide a calibration process of merging, into one spatial dimension, all of a calibration task required when each camera generates a depth map and the generated depth map of each camera in an inspection optical system including multiple cameras and a projector.

Description

Multi-camera 3D correction device and method based on multiple Z-axis moving grid patterns

[0001] The technical concept of the present disclosure relates to a multi-camera 3D calibration device and method based on a multi-Z-axis moving grid pattern, and more specifically, to a calibration device and method for performing a calibration operation required when generating a depth map from each camera within an inspection optical system composed of multiple cameras and a projector, and merging all depth maps generated from each camera into a single spatial dimension.

[0002] Unless otherwise indicated in this specification, the contents described in this section are not prior art for the claims of this application, and are not to be recognized as prior art simply because they are included in this section.

[0003] Multi-camera 3D calibration is the process of adjusting and correcting the position, angle, and lens distortion of each camera to accurately capture and analyze the same 3D space using multiple cameras. Multi-camera 3D calibration plays a crucial role in combining scenes captured simultaneously by multiple cameras to generate consistent 3D information and minimizing distortion that may occur from the individual perspectives of each camera. Through multi-camera 3D calibration, the relative positions and orientations of each camera are aligned to accurately reconstruct objects in 3D space captured by multiple cameras. This ensures that the scenes captured by multiple cameras fit together precisely to appear as a single, integrated 3D scene. Furthermore, camera lenses generally exhibit distortion, which is particularly pronounced in lenses with wide fields of view; correcting this distortion ensures consistency in the images captured by each camera. Additionally, in multi-camera systems, all cameras shoot simultaneously or capture images at consistent timings to obtain accurate 3D data and depth information.

[0004] In conventional multi-camera 3D calibration methods, two types of jigs were used for accurate 3D calibration. Jigs are tools that play a crucial role in camera calibration, used to fix specific parts or equipment or to precisely align their positions. In multi-camera systems, since cameras are positioned at different angles and locations, it is important to accurately establish their spatial positional relationships. To this end, two types of jigs are typically required, each used for a different purpose. The first jig is used to precisely align the individual position and angle of each camera, ensuring that all cameras are correctly aligned with the same reference point. The second jig is used to adjust the relative positional relationships between the multiple cameras, thereby enabling each camera to perceive 3D space in a consistent manner.

[0005] In conventional Phase Measuring Profilometry (PMP), height data is calculated from each pixel of the camera. However, it is often necessary to calculate 3D point cloud data based on this height data. Furthermore, existing multi-camera 3D calibration methods have the disadvantage of being time-consuming and complex because they require performing the calibration process individually using the two aforementioned jigs. Additionally, multi-camera 3D calibration necessitates the precise placement and individual calibration of multiple cameras. Since this process requires multiple adjustments and tests, it consumes a significant amount of time. Moreover, installing and adjusting jigs or patterns individually to calibrate each camera takes a considerable amount of time. In particular, as the number of cameras increases, this process becomes even more complex and difficult to manage.

[0006] Prior art documents include Korean Patent Registration No. 10-2166236 (October 8, 2020) and Korean Patent Registration No. 10-1763855 (July 26, 2017).

[0007] Previously, in order to perform multi-camera 3D calibration precisely, it was necessary to use two types of jigs and perform a separate process for each jig. To solve this, the technical concept of the present disclosure enables the image calibration process to be performed at once and precision to be maintained through a multi-camera 3D calibration device and method based on multiple Z-axis moving grid patterns.

[0008] In addition, the multi-camera 3D calibration device and method based on a multi-Z-axis moving grid pattern according to the embodiment provides a calibration process that is required when generating a depth map from each camera within an inspection optical system composed of multiple cameras and a projector, and a calibration process that merges all generated depth maps from each camera into a single spatial dimension.

[0009] In addition, in the embodiment, a correction jig engraved with a dot is positioned on a reference plane (z = 0) within the inspection optical system, and a moving grid pattern projected from a projector within the inspection optical system is captured by each camera. Subsequently, the phase of the reference plane (z = 0) is determined using the moving grid pattern information within the captured image, and the perspective matrix at the reference plane (z = 0) is calculated using the dot information.

[0010] Afterwards, the calibration jig is given a height change (z ≠ 0) using the Z-axis, and the phase and projection matrix at that height are obtained in the manner described above. This process is repeated N times (z = _1, _2, … _), and the amount of change along the Z-axis moves constantly during each repetition.

[0011] In the embodiment, the calibration required to obtain constants when generating depth maps from each camera can be obtained through the difference between the reference plane phase and the phase measured for each height change, and the calibration required to obtain constants when merging depth maps generated from each camera into one spatial dimension can be obtained through the perspective matrix obtained at each height.

[0012] However, the problem to be solved according to one embodiment is not limited only to that mentioned above.

[0013] A multi-camera 3D correction method based on a multi-Z-axis moving grid pattern according to an embodiment may include: a step of placing a correction jig having dots arranged in a matrix at regular intervals on a jig for photographing an inspection target, and generating a correction data set by photographing the dots arranged in a matrix on the correction jig while sequentially moving the Z-axis of the correction jig at regular intervals without an inspection target; a step of calculating a calibration constant for generating a depth map using the generated data set and storing the calculated calibration constant in a database; a step of photographing a 3D image of the inspection target when the inspection target is placed on a device; and a step of generating 3D image data by applying the calibration constant to the photographed image.

[0014] Additionally, the step of generating 3D image data may include: a step of capturing an image of an object to be inspected after investigating a moving grid pattern; a step of generating a depth map using a calibration constant for generating a depth map; a step of converting the generated depth map into a point cloud; a step of converting the converted point cloud into a calibration depth map of each camera by inversely applying a perspective matrix of the reference camera; and a step of generating 3D data by merging the calibration depth maps.

[0015] Additionally, the step of generating a correction data set may include: performing a moving grid pattern inspection and capturing an image of the correction jig without moving the Z-axis of the correction jig at each camera included in the optical system, and then repeating the moving grid pattern inspection and image capturing n times after moving the Z-axis of the correction jig stepwise.

[0016] In addition, the step of calculating a calibration constant for generating a depth map using the generated dataset and storing the calculated calibration constant in a database involves calculating a reference phase where the height (h) is 0 without moving the Z-axis and a perspective matrix of the reference phase, measuring the phase whenever there is a change in height using moving grid pattern information, calculating a calibration constant value using the difference between the phase at each height and the reference phase, and the calibration constant value may be a constant that converts the phase difference into height.

[0017] In addition, the step of calculating calibration constants for depth map generation using the generated dataset and database-ing the calculated calibration constants can be performed by using dot information to obtain a perspective matrix at each height and database-ing the perspective matrix for each height.

[0018] In addition, the step of calculating calibration constants for depth map generation using the generated dataset and storing the calculated calibration constants in a database involves, given N image coordinates and real-world coordinates corresponding to each of the image coordinates, a mathematical formula

[0019]

[0020] We calculate the parameter pi (i=1 to 8) that most accurately satisfies the least square error, and if N=4, a unique solution exists, and if N>4, it becomes an overdetermined linear equation, so a solution can be calculated through the pseudo-inverse method.

[0021] In addition, the step of converting the generated depth map into a point cloud can convert the depth map data into point cloud data using the measured height and the neighboring lookup table values ​​P (p1 to p8).

[0022] Previously, separate calibration processes had to be performed using two jigs, but the multi-Z-axis moving grid pattern-based multi-camera 3D calibration device and method according to the embodiment can simplify the process and save time by performing multi-camera 3D calibration at once using a moving grid pattern.

[0023] In addition, the embodiment enables precise alignment of depth maps generated from each camera through correction based on multiple Z-axis moving grid patterns, thereby allowing for better correction results while maintaining precision during the correction process.

[0024] In addition, the embodiment includes a process of merging depth maps generated from each camera into a single spatial dimension, which reduces errors occurring during the merging process and increases the consistency of depth information. This allows for a significant improvement in the accuracy of the inspection optical system.

[0025] In addition, in the embodiment, the correction process at each height can be optimized by utilizing the moving grid pattern and dot information to obtain a projection matrix based on the reference plane and height change. This enables more accurate 3D correction by taking into account variations due to height changes.

[0026] In addition, in the embodiment, correction can be performed repeatedly while moving the Z-axis at a constant rate, thereby enabling more automated correction, which allows for repetitive and consistent correction while saving time and manpower.

[0027] The aforementioned effects maximize the efficiency of multi-camera 3D correction and contribute to improving the accuracy and reliability of the system.

[0028] The effects obtainable from the exemplary embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure.

[0029] FIG. 1 is a drawing showing an optical system according to an embodiment.

[0030] FIG. 2 is a drawing for explaining the depth map and point cloud injected in an embodiment.

[0031] FIG. 3 is a drawing for explaining the height measurement process according to an embodiment.

[0032] FIG. 4 is a drawing showing the configuration of a multi-camera 3D correction device (100) based on a multi-Z-axis moving grid pattern according to an embodiment.

[0033] FIG. 5 is a drawing showing a calibration plate according to an embodiment.

[0034] FIG. 6 is a drawing illustrating the process of converting point clouds collected from each camera of a 3D correction device according to an embodiment into correction depth maps and merging a plurality of converted correction depth maps to generate 3D data for an inspection target.

[0035] FIG. 7 is a diagram illustrating an interpolation process for correcting a different height when a different height is measured in addition to n heights already measured in a 3D correction device according to an embodiment.

[0036] FIG. 8 is a diagram illustrating the process of generating a projection matrix lookup table according to an embodiment.

[0037] FIG. 9 is a drawing for explaining the 3D data generation process according to an embodiment.

[0038] FIG. 10 is a drawing showing a Perspective Transform Calibration Example according to an embodiment.

[0039] FIG. 11 is a drawing showing the projection transformation correction result according to an embodiment.

[0040] FIG. 12 is a drawing showing corrected coordinates according to an embodiment.

[0041] Hereinafter, various embodiments of the present disclosure are described in conjunction with the accompanying drawings. As various embodiments of the present disclosure may be subject to various modifications and may have various forms, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the various embodiments of the present disclosure to specific forms, and it should be understood that they include all modifications and / or equivalents and substitutions that fall within the spirit and scope of the various embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals have been used for similar components.

[0042] In various embodiments of the present disclosure, terms such as “comprising” or “having” are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0043] In various embodiments of the present disclosure, expressions such as “or” include any and all combinations of the words listed together. For example, “A or B” may include A, may include B, or may include both A and B.

[0044] Expressions such as "first," "second," "first," or "second" used in various embodiments of the present disclosure may modify various components of the various embodiments, but do not limit such components. For example, such expressions do not limit the order and / or importance of such components and may be used to distinguish one component from another.

[0045] When it is mentioned that a component is "connected" or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that a new component may also exist between the component and the other component.

[0046] In the embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are used to refer to a component that performs at least one function or operation, and such component may be implemented in hardware or software, or in a combination of hardware and software. Additionally, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except where each needs to be implemented in specific individual hardware.

[0047] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the various embodiments of the present disclosure.

[0048] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0049] FIG. 1 is a diagram showing an optical system according to an embodiment.

[0050] Referring to FIG. 1, in the embodiment, a multi-Z-axis moving grid pattern-based multi-camera 3D calibration device (100) performs calibration work required when generating a depth map in each camera (10, 20, 30) included in an inspection optical system composed of multiple cameras and a projector, and performs calibration work to merge all depth maps generated by each camera into one spatial dimension.

[0051] FIG. 2 is a drawing for explaining the depth map and point cloud injected in the embodiment.

[0052] Referring to FIG. 2, a depth map is a map containing depth information for each pixel of an image. It is generally represented as a grayscale image, where the brightness of the pixel indicates the depth value. Higher brightness indicates a closer distance, while darker brightness indicates a farther distance. As illustrated in FIG. 2, the depth map represents the distance from the camera to each pixel and consists solely of distance data (z).

[0053] A point cloud is a set of individual points in 3D space, where each point has position coordinates (x, y, z) at a specific point in space. These points are typically collected through laser scanners, LiDAR, 3D cameras, etc., and are used to represent the surface of real objects or environments. As illustrated in FIG. 2, a point cloud represents coordinate data within a specific coordinate system and consists of coordinate data (x, y, z).

[0054] In the embodiment, the 3D correction device (100) uses height data h(i, j) at a given pixel (i, j) to find the world coordinates (x, y) corresponding to the pixel (i, j) in z=h(i, j) in order to convert the depth map, which is an image coordinate, into spatial coordinates.

[0055] To this end, in the embodiment, a calibration plate containing dots is sequentially moved from the position z=h1 (h1=0) to the position z=hn. Here, n is the number of calibration planes for height. FIG. 5 is a diagram showing a calibration plate according to the embodiment. As shown in FIG. 5, the calibration plate according to the embodiment includes a plurality of dots spaced apart at regular intervals. Subsequently, at each plate position z=hi, matrix calculations for the projection transformation between image coordinates and world coordinates on the calibration plane are performed. FIG. 3 is a diagram illustrating the height measurement process according to the embodiment.

[0056] In addition, in the embodiment, two adjacent correction planes z=hl and z=hu are found, and hl <h<hu의 조건을 만족하도록 한다. 픽셀(i, j)에 대응하는 z=hl 평면에서의 월드 좌표 (xl, yl)과 z=hu평면에서의 월드 좌표 (xu, yu)를 계산한다. 이후, (xl, yl) 와 (xu, yu)의 선형 보간을 통해 z=h 에서의 x, y를 계산한다.

[0057] FIG. 4 is a diagram showing the configuration of a multi-camera 3D correction device (100) based on a multi-Z-axis moving grid pattern according to an embodiment.

[0058] The configuration of the multi-camera 3D correction device (100) based on a multi-Z-axis moving grid pattern shown in FIG. 4 is merely a simplified example. The communication module (110) can be configured regardless of the communication mode, such as wired or wireless, and can be configured with various communication networks, such as a Personal Area Network (PAN) or a Wide Area Network (WAN). Additionally, the communication module (110) can operate based on the known World Wide Web (WWW) and may utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. For example, the communication module (110) may be responsible for transmitting and receiving data necessary to perform a technique according to one embodiment of the present disclosure.

[0059] Memory (120) may refer to any type of storage medium. For example, memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Such memory (120) may also constitute the database shown in FIG. 1.

[0060] Memory (120) can store at least one instruction that can be executed by the processor (130). Additionally, memory (120) can store any form of information generated or determined by the processor (130) and any form of information received by the server (200). For example, memory (120) stores RM data and RM protocols according to the user, as will be described later. Additionally, memory (120) stores various types of modules, instruction sets, or models.

[0061] The processor (130) can perform technical features according to embodiments of the present disclosure to be described below by executing at least one instruction stored in memory (120). In one embodiment, the processor (130) may be composed of at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU) of a computer device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0062] This processor (130) can train a neural network or model designed in a machine learning or deep learning manner. To this end, the processor (130) can perform calculations for training the neural network, such as processing input data for training, extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. Additionally, the processor (130) can perform inference for a specific purpose using a model implemented in an artificial neural network manner.

[0063] In the embodiment, the processor (130) places a correction jig, on which a pattern is matrix-arranged at regular intervals on a jig for photographing an inspection target, into the device, and generates a correction data set by photographing the pattern matrix-arranged on the correction jig while sequentially moving the Z-axis of the correction jig at regular intervals without an inspection target. In the embodiment, the pattern may be formed through a plurality of dots matrix-arranged at regular intervals on the jig, and the object forming the pattern according to the embodiment is not limited to a dot.

[0064] Accordingly, in the embodiment, for multi-camera 3D correction based on a multi-Z-axis moving grid pattern, a correction jig having dots arranged in a matrix at regular intervals is placed on a jig for photographing an inspection target, and a correction data set is generated by photographing the dots arranged in a matrix on the correction jig while sequentially moving the Z-axis of the correction jig at regular intervals without an inspection target.

[0065] In the embodiment, a correction jig engraved with a dot is positioned on a reference plane (z = 0) within the inspection optical system, and at this time, a moving grid pattern projected from a projector within the inspection optical system is captured by each camera. The phase of the reference plane (z = 0) is determined using the moving grid pattern information within the captured image, and the perspective matrix at the reference plane (z = 0) is determined using the dot information. Subsequently, the correction jig is given a height change (z ≠ 0) using the Z-axis, and the respective phase and perspective matrix at that height are determined in the same manner as above. This process is repeated N times (z = X_1, X_2, … X_n), and the amount of change along the Z-axis moves constantly during the repetition.

[0066] In the embodiment, the calibration required to obtain the constants for generating depth maps from each camera can be obtained through the difference between the reference plane phase and the phase measured for each height change, and the calibration required to obtain the constants for merging the depth maps generated from each camera into a single spatial dimension can be obtained through the perspective matrix obtained at each height. Previously, precise multi-camera 3D calibration required separate processes using two types of jigs, but the embodiment enables this to be done through a single process while maintaining precision. That is, in the embodiment, both types of calibration required for 3D data restoration are performed through a single process in which an image is captured for each camera whenever there is a height change using the Z-axis.

[0067] To this end, the processor (130) performs image capture after investigating a moving grid pattern without moving the Z-axis in each camera included in the optical system, and repeats the pattern investigation and image capture after moving the Z-axis n times. In the embodiment, image capture is performed in each camera within the optical system and n times according to a set height. For example, if four cameras are installed in the optical system (front, back, left, and right), n times of capture are performed in each camera corresponding to the front, back, left, and right. Afterward, the processor (130) calculates a calibration constant for generating a depth map.

[0068] The processor (130) calculates a reference phase and a perspective matrix of the reference phase where the height (h) without shifting the Z-axis is 0, in order to calculate a calibration constant for generating a depth map. At this time, the processor (130) measures the phase whenever there is a change in height using the moving grid pattern information, and calculates a calibration constant value using the difference between the phase at each height and the reference phase. In the embodiment, the calibration constant value is a constant that converts the phase difference into height.

[0069] In the embodiment, the processor (130) calculates a calibration constant for generating a depth map using the generated data set and stores the calculated calibration constant in a database. Subsequently, when the inspection target is placed on the device, a 3D image of the inspection target is captured, and 3D image data is generated by applying the calibration constant to the captured image. In the embodiment, the processor (130) calculates calibration coefficients a and b for all pixels using the difference between the phases measured while moving along the Z-axis (height change) and the phase of the reference plane. In the embodiment, the processor (130) can calculate the calibration coefficients through Equation 1.

[0070]

[0071] Subsequently, the processor (130) estimates the parameters a(i, j) and b(i, j) through Equation 1 and completes the correction by implementing the subsequent procedure. The distribution of parameters a(i, j) and b(i, j) across the entire image can be used to set up a lookup table (LUT). Then, when the phase difference distribution of the object is measured in an actual measurement, the corresponding depth map can be reconstructed through Equation 2.

[0072]

[0073] In mathematical equation 2, ψ(I, j) represents the phase difference.

[0074] Additionally, the processor (130) obtains a perspective matrix at each height using dot information and stores the perspective matrix for each height in a database. In the embodiment, the perspective matrix for each height can be stored as a lookup table.

[0075] In the embodiment, camera correction is a transformation process between the image coordinate system and the real-world coordinate system. Camera correction compensates for distortion for accurate measurement in real-world coordinates, and corrects for the difference in aspect ratio of non-square pixels in the image coordinate system. Additionally, it corrects for optical distortion of the camera lens. The process of calculating the projection matrix is ​​described below.

[0076] In an embodiment, the processor (130) calculates a calibration constant for generating a depth map when given N image coordinates and real-world coordinates corresponding to each of the image coordinates,

[0077]

[0078] A parameter pi (i=1 to 8) that most accurately satisfies the least square error is calculated. In the example, if N=4, a unique solution exists, and if N>4, it becomes an overdetermined linear equation, so a solution is calculated using the pseudo-inverse method.

[0079] A pseudo-inverse is a matrix used when a general inverse matrix does not exist, and in the example, it can be calculated according to Equation 4.

[0080]

[0081] The equation AP=b in mathematical formula 4 can be expressed as a matrix as follows.

[0082]

[0083] Subsequently, the processor (130) generates a depth map using a correction constant for generating a depth map and converts the generated depth map into a point cloud. At this time, the processor (130) converts the depth map data into point cloud data using the measured height and the values ​​P(p1, p2, ... p8) of the adjacent lookup table. The point cloud conversion process can be explained through Equation 5.

[0084]

[0085] In mathematical equation 5, (I, J) is an image coordinate as a depth map coordinate, and (x, y) is a 3D coordinate as a point cloud coordinate.

[0086] Afterward, the processor (130) converts the converted point cloud into a correction depth map of each camera by applying the projection matrix of the reference camera inversely.

[0087] In the embodiment, the reference camera refers to a main camera that photographs the inspection target vertically, and

[0088] The data format is depth map data, but the coordinates are converted to the image coordinate system of the reference camera. Subsequently, the processor (130) merges the converted correction depth maps of each camera to generate 3D data. FIG. 6 is a diagram illustrating the process of converting point clouds collected from each camera of a 3D correction device according to an embodiment into correction depth maps and merging a plurality of converted correction depth maps to generate 3D data for an inspection target.

[0089] FIG. 7 is a diagram illustrating the interpolation process for correction at a different height when a height other than the n heights already measured in the 3D correction device according to the embodiment is measured. Referring to FIG. 7, while there are projection matrices for a height h1 of 100 μm and a height h2 of 200 μm, if the measured height is 150 μm, the coordinates corrected at a height of 150 μm can be calculated through interpolation using the coordinates transformed at height h1 and the coordinates transformed at height h2. For example, the processor (130) calculates the coordinates (x_h2, y_h2) transformed using P_h2 and the coordinates (x_h1, y_h2) transformed using P_h1. Subsequently, the processor (130) calculates (x_h, y_h) by linearly interpolating each, taking into account the position of the h value. The corrected position of the h value can be calculated through Equation 6.

[0090]

[0091] Hereinafter, we will look at FIG. 8. The multi-Z axis moving grid pattern-based multi-camera 3D correction method illustrated in FIG. 8 can be performed by a multi-Z axis moving grid pattern-based multi-camera 3D correction device (100) including a processor (130).

[0092] Meanwhile, FIG. 8 is merely illustrative, and the concept of the present invention is not to be interpreted as being limited to that shown in FIG. 8. For example, each step may be configured in a different order than that shown in FIG. 8, at least one of the steps shown in FIG. 8 may not be performed, or one or more steps not shown in FIG. 8 may be additionally performed.

[0093] Below, a multi-camera 3D correction method based on a multi-Z-axis moving grid pattern will be described in turn. Since the operation (function) of the multi-camera 3D correction method based on a multi-Z-axis moving grid pattern according to the embodiment is essentially the same as the function of the system, descriptions that overlap with FIGS. 1 to 7 will be omitted.

[0094] FIG. 8 is a diagram illustrating the process of generating a projection matrix lookup table according to an embodiment. Referring to FIG. 8, in step S110, an image is captured after examining a moving grid pattern on a dot chart. In the embodiment, in step S110, the pattern examination and capture are performed after moving the Z-axis (h=1). The process of examining and capturing the pattern while moving the Z-axis can be repeated n times depending on the height of the Z-axis. In the embodiment, pattern examination and capture are performed while moving the Z-axis (h=n). Subsequently, the process proceeds to step S120 to calculate calibration constants for generating a depth map and to generate a perspective matrix lookup table. Once the lookup table is generated, the projection matrix is ​​used after capturing the object to be inspected. This process is explained through FIG. 9. FIG. 9 is a diagram illustrating the process of generating 3D data according to an embodiment.

[0095] Referring to FIG. 9, in step S130, an image is captured after irradiating the inspection target with a moving grid pattern. At this time, a depth map based on each camera that captured the inspection target is generated using a correction constant for generating the depth map. Subsequently, in step S140, the depth map is converted into a point cloud using a projection matrix lookup table. Subsequently, in step S150, the projection matrix is ​​inversely applied from the projection matrix lookup table of the reference camera to convert it into a depth map based on the reference camera. Subsequently, in step S160, the converted depth maps are merged to generate 3D data.

[0096] FIG. 10 is a diagram showing a Perspective Transform Calibration Example according to an embodiment, FIG. 10 is a diagram showing the result of the Perspective Transform Calibration according to an embodiment, and FIG. 12 is a diagram showing the calibrated coordinates according to an embodiment. In the embodiment, a calibrated image as shown in FIG. 10 can be obtained by performing the calibration work required when generating a depth map at each camera.

[0097] Previously, separate calibration processes had to be performed using two jigs, but the multi-Z-axis moving grid pattern-based multi-camera 3D calibration device and method according to the embodiment can simplify the process and save time by performing multi-camera 3D calibration at once using a moving grid pattern.

[0098] In addition, the embodiment enables precise alignment of depth maps generated from each camera through correction based on multiple Z-axis moving grid patterns, thereby allowing for better correction results while maintaining precision during the correction process.

[0099] In addition, the embodiment includes a process of merging depth maps generated from each camera into a single spatial dimension, which reduces errors occurring during the merging process and increases the consistency of depth information. This allows for a significant improvement in the accuracy of the inspection optical system.

[0100] In addition, in the embodiment, the correction process at each height can be optimized by utilizing the moving grid pattern and dot information to obtain a projection matrix based on the reference plane and height change. This enables more accurate 3D correction by taking into account variations due to height changes.

[0101] In addition, in the embodiment, correction can be performed repeatedly while moving the Z-axis at a constant rate, thereby enabling more automated correction, which allows for repetitive and consistent correction while saving time and manpower.

[0102] The aforementioned effects maximize the efficiency of multi-camera 3D correction and contribute to improving the accuracy and reliability of the system.

[0103] Meanwhile, the methods according to the various embodiments of the present invention described above can be implemented in the form of an application or software program that can be installed on an existing electronic device.

[0104] In addition, the whole or part of the method may be composed of multiple software function modules and implemented on an operating system (OS). Alternatively, each step may be composed of a single software function module, or each step may be combined to form a single software function module and implemented on an operating system. Therefore, even if all of the embodiments of the present disclosure are not implemented as a single software function module, if multiple software function modules implement each step of the present disclosure and multiple software function modules are implemented on a single operating system, it can be understood that the method of the present disclosure has been implemented.

[0105] Furthermore, the methods according to the various embodiments of the present invention described above may be implemented solely through software upgrades or hardware upgrades of existing electronic devices. Additionally, the various embodiments of the present invention described above may also be performed through an embedded server equipped in an electronic device or through an external server of the electronic device.

[0106] Meanwhile, according to one embodiment of the present invention, the various embodiments described above may be implemented as software comprising instructions stored on a computer-readable recording medium using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0107] Meanwhile, a computer or a similar device may include a device according to the disclosed embodiments, which is capable of calling instructions stored from a storage medium and operating according to the called instructions. When said instructions are executed by a processor, the processor may perform a function corresponding to said instructions directly or by using other components under the control of said processor. The instructions may include code generated or executed by a compiler or an interpreter.

[0108] A computer-readable recording medium may be provided in the form of a non-transitory computer-readable recording medium. Here, "non-transitory" simply means that the storage medium does not contain a signal and is tangible, without distinguishing whether data is stored semi-permanently or temporarily on the storage medium. In this context, a non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short moment, such as registers, caches, or memory. Specific examples of non-transitory computer-readable media may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0109] As described above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used only for the purpose of explaining the technical concept of this disclosure and are not intended to limit the meaning or the scope of this disclosure as defined in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of this disclosure should be determined by the technical concept of the appended claims.

[0110] The present disclosure is applicable to the field of multi-camera 3D correction technology based on multiple Z-axis moving grid patterns.

Claims

1. In a multi-camera 3D correction method based on multiple Z-axis moving grid patterns, A step of placing a correction jig, having a pattern matrix-arranged at regular intervals on a jig for photographing an inspection target, into a device, and generating a correction data set by photographing the pattern matrix-arranged on the correction jig while sequentially moving the Z-axis of the correction jig at regular intervals without an inspection target; A step of calculating calibration constants for depth map generation using the above-mentioned generated dataset and storing the calculated calibration constants in a database; When the inspection target is placed on the device, a step of capturing a 3D image of the inspection target; A 3D correction method comprising the step of generating 3D image data by applying the correction constant to the captured image.

2. In claim 1, the step of generating the 3D image data A step of capturing an image of the inspection target after investigating the moving grid pattern; A step of generating a depth map using a calibration constant for generating a depth map; A step of converting the depth map generated above into a point cloud; A step of converting the above-mentioned converted point cloud into a correction depth map of each camera by inversely applying the height-dependent projection matrix (Perspective Matrix) of the reference camera; and A 3D correction method comprising the step of generating 3D data by merging the above correction depth maps.

3. In paragraph 1, the step of generating the correction data set A 3D correction method comprising: a step of, in each camera included in an optical system, performing a moving grid pattern inspection and capturing an image of a correction jig without moving the Z-axis of the correction jig, and then repeating the moving grid pattern inspection and image capturing n times after moving the Z-axis of the correction jig in stages.

4. In paragraph 2, the step of calculating a calibration constant for depth map generation using the generated data set and storing the calculated calibration constant in a database Calculate a reference phase with a height (h) of 0 that is not shifted along the Z-axis and a perspective matrix of the reference phase, and Using moving grid pattern information, the phase is measured whenever there is a change in height, and a calibration constant value is calculated using the difference between the phase at each height and the reference phase. A 3D correction method in which the above correction constant value is a constant that converts the phase difference into height.

5. In claim 1, the step of calculating a calibration constant for depth map generation using the generated data set and storing the calculated calibration constant in a database A 3D correction method that calculates a perspective matrix at each height using pattern information and databases the perspective matrices for each height.

6. In paragraph 4, the step of calculating a calibration constant for depth map generation using the generated data set and storing the calculated calibration constant in a database Given N image coordinates and real-world coordinates corresponding to each of the image coordinates, a mathematical formula Calculate the parameter pi (i=1 to 8) that most accurately satisfies the least squares error, and A 3D correction method in which a unique solution exists when N=4, and an overdetermined linear equation is obtained when N>4, using a pseudo-inverse method.

7. In paragraph 2, the step of converting the generated depth map into a point cloud A 3D correction method that converts depth map data into point cloud data using P (p1 to p8) of the measured height and adjacent lookup table values.

8. Memory storing at least one instruction for multi-camera 3D correction based on multiple Z-axis moving grid patterns; and It includes a processor that performs an operation according to the above instruction, The above processor is, A step of generating a correction data set by sequentially moving the Z-axis of the correction jig at regular intervals while capturing the matrix-arranged pattern on the correction jig while the inspection target is placed on a jig for photographing the inspection target; Using the above-mentioned generated dataset, calibration constants for depth map generation are calculated, and the calculated calibration constants are stored in a database. When the object to be inspected is placed on the device and a 3D image of the object to be inspected is captured, A 3D correction device that generates 3D image data by applying the correction constant to the above-described image.

9. In paragraph 8, the above processor After examining the moving grid pattern, capture an image of the inspection target, and Generate a depth map using calibration constants for depth map generation, and Convert the depth map generated above into a point cloud, and The above-mentioned transformed point cloud is converted into a corrected depth map for each camera by inversely applying the perspective matrix of the reference camera, and A 3D correction device that generates 3D data by merging the above correction depth maps.

10. In paragraph 8, the processor A 3D correction device that, in each camera included in the optical system, performs irradiating a moving grid pattern and capturing an image of a correction jig without moving the Z-axis of the correction jig, and then repeats irradiating the moving grid pattern and capturing an image n times after moving the Z-axis of the correction jig in stages.

11. In paragraph 9, the processor Calculate a reference phase with a height (h) of 0 that is not shifted along the Z-axis and a perspective matrix of the reference phase, and Using moving grid pattern information, the phase is measured whenever there is a change in height, and a calibration constant value is calculated using the difference between the phase at each height and the reference phase. A 3D correction method in which the above correction constant value is a constant that converts the phase difference into height.

12. In Clause 10, the above processor A 3D correction device that calculates a perspective matrix at each height using pattern information and databases the perspective matrices for each height.

13. In paragraph 11, the above processor Given N image coordinates and real-world coordinates corresponding to each of the image coordinates, a mathematical formula Calculate the parameter pi (i=1 to 8) that most accurately satisfies the least squares error, and A 3D correction device that, if N=4, a unique solution exists, and if N>4, it becomes an overdetermined linear equation and calculates a solution through a pseudo-inverse method.

14. In paragraph 9, the processor A 3D correction device that converts depth map data into point cloud data using the measured height and the adjacent lookup table values ​​P (p1 to p8).

Images