Marker pose uncertainty estimation method and system

Abstract

One embodiment of the present invention provides a marker pose uncertainty estimation method. The method comprises the steps of: receiving a single image that includes a reference marker, and deriving a corner position for the reference marker on the basis of a pixel intensity variation of the reference marker; deriving corner position uncertainty for the corner position on the basis of the pixel intensity variation, and deriving the position and the pose of the reference marker on the basis of the corner position; and deriving the position and the pose uncertainty of the reference marker on the basis of the corner position and the corner position uncertainty.

Description

Method and System for Estimating Marker Posture Uncertainty

[0001] The present invention relates to a method and system for estimating marker pose uncertainty, and more specifically, to a method and system for estimating marker pose uncertainty that receives a single image including a reference marker, derives a corner position and corner position uncertainty for the reference marker based on a change in pixel intensity of the reference marker, and derives a position and pose uncertainty of the reference marker based on the corner position and said corner position uncertainty.

[0002] Fiducial markers are utilized in fields such as manufacturing process automation and medical image analysis. In manufacturing process automation, fiducial markers are printed on parts, allowing the manufacturing process system to locate and position the parts by recognizing them. In medical image analysis, fiducial markers are displayed on surgical sites and equipment, enabling the image analysis system to identify the location of these sites and equipment by recognizing the displayed markers.

[0003] Since reference markers serve as reference locations for specific targets in manufacturing processes and medical image analysis, the position and orientation of the reference marker must be accurately identified in images acquired through cameras, etc.

[0004] Conventional reference marker recognition technology acquires an image containing a reference marker through an alignment camera and derives the position and orientation of the reference marker by detecting its corner location within the image. However, since the images acquired by conventional reference marker recognition technology contain reference markers that have suffered information loss due to the alignment camera's image resolution, noise, lens distortion, and quantization, the accuracy of deriving the position and orientation of the reference marker is currently low because the precise corner location of the reference marker cannot be detected.

[0005] The present invention aims to solve the aforementioned problems and has as its technical objective to provide a method and system for estimating marker pose uncertainty, which receives a single image including a reference marker, derives a corner position and corner position uncertainty for the reference marker based on a change in pixel intensity of the reference marker, and derives a 6-degrees-of-freedom uncertainty of the reference marker based on the corner position and the corner position uncertainty.

[0006] The technical problems that the present invention aims to solve are not limited to the technical problems described above, and other technical problems of the present invention may be derived from the following description.

[0007] As a technical means for solving the aforementioned technical problem, according to one aspect of the present invention, a method for estimating marker pose uncertainty is provided. The method comprises the steps of receiving a single image including a reference marker and deriving a corner position for the reference marker based on a change in pixel intensity of the reference marker; deriving a corner position uncertainty for the corner position based on the change in pixel intensity and deriving a position and pose of the reference marker based on the corner position; and deriving a position and pose uncertainty of the reference marker based on the corner position and the corner position uncertainty.

[0008] In addition, as a technical means for solving the aforementioned technical problem, according to another aspect of the present invention, a marker pose uncertainty estimation system is provided. The system comprises at least one processor and a memory electrically connected to the processor and storing a marker pose uncertainty estimation program executed by the processor, wherein the memory stores code that causes the processor to receive an image including a reference marker, derive a corner position for the reference marker based on a change in pixel intensity of the reference marker, derive a corner position uncertainty for the corner position based on the change in pixel intensity, derive a position and pose of the reference marker based on the corner position, and derive a position and pose uncertainty of the reference marker based on the corner position and the corner position uncertainty.

[0009] According to the means for solving the problem of the present invention described above, the position and orientation of a reference marker and the uncertainty regarding the position and orientation of the reference marker can be derived, thereby increasing the reliability of the derived position and orientation of the reference marker.

[0010] In addition, since reference marker pose uncertainty is derived using computer vision algorithms and non-linear state estimation algorithms, the amount of computation can be significantly reduced compared to AI model-based marker recognition technology that requires a large amount of training data.

[0011] In addition, the position and orientation of the reference marker and the uncertainty regarding the position and orientation of the reference marker can be derived from a single image containing the reference marker.

[0012] FIG. 1 is a block diagram illustrating the configuration of a marker attitude uncertainty estimation system according to one embodiment of the present invention.

[0013] Figure 2 is a diagram showing an example of deriving the position and attitude uncertainty of a reference marker by the marker attitude uncertainty estimation system illustrated in Figure 1.

[0014] Figure 3 is a diagram showing an example of generating the corner position and expected corner point of a reference marker in a single image by the marker pose uncertainty estimation system illustrated in Figure 1.

[0015] FIG. 4 is a flowchart illustrating a method for estimating marker pose uncertainty according to another embodiment of the present invention.

[0016] Figure 5 is a flowchart showing the detailed steps included in the steps of the marker pose uncertainty estimation method illustrated in Figure 4.

[0017] The present invention will be described in detail below with reference to the attached drawings. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification, and the technical concept disclosed in this specification is not limited by the attached drawings. All terms used herein, including technical and scientific terms, should be interpreted in the sense generally understood by those skilled in the art to which the present invention pertains. Terms defined in advance should be interpreted as having additional meanings consistent with relevant technical literature and the presently disclosed content, and should not be interpreted in a highly ideal or restrictive sense unless otherwise defined.

[0018] In order to clearly explain the invention in the drawings, parts unrelated to the explanation have been omitted, and the size, shape, and form of each component shown in the drawings may be varied. Throughout the specification, identical or similar parts are denoted by identical or similar reference numerals.

[0019] Suffixes such as "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, detailed descriptions of related prior art have been omitted where it is determined that such detailed descriptions could obscure the essence of the embodiments disclosed in this specification.

[0020] Throughout the specification, when it is stated that a part is "connected (connected, contacted, or coupled)" to another part, this includes not only cases where they are "directly connected (connected, contacted, or coupled)," but also cases where they are "indirectly connected (connected, contacted, or coupled)" with other members interposed therebetween. Furthermore, when it is stated that a part "includes (provides, or provides)" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but rather allows for additional "included (provided, or provided)" of other components.

[0021] The singular forms of expression used in this specification should be interpreted as including plural forms of expression unless the opposite meaning is clearly indicated.

[0022] The position and orientation described below may be six degrees of freedom (6DOF) position and orientation. For example, the position of the reference marker is a position in three-dimensional space (x, y, z), and the orientation of the reference marker is a state in which the reference marker is rotated or tilted in a specific direction, which may mean a state in which it is rotated or tilted around the x-axis, y-axis, and z-axis.

[0023] FIG. 1 is a block diagram illustrating the configuration of a marker attitude uncertainty estimation system (100) according to one embodiment of the present invention, and FIG. 2 is a diagram showing an example of deriving the position and attitude uncertainty of a reference marker by the marker attitude uncertainty estimation system.

[0024] Referring to FIGS. 1 and 2, the marker pose uncertainty estimation system (100) includes at least one processor (140) and memory (150), and may further include a communication module (110), a camera module (120), and a database (130). The marker pose uncertainty estimation system (100) derives a reference marker pose uncertainty using a computer vision algorithm and a non-linear state estimation algorithm.

[0025] The communication module (110) can transmit and receive data necessary for estimating marker pose uncertainty by performing information transmission and reception with an external device or server. The camera module (120) can acquire an image including a reference marker.

[0026] The database (130) may be a place where data necessary for private information detection and masking is stored. The database (120) may be built in a part of the memory (140) or implemented as separate hardware.

[0027] The processor (140) performs operations according to the code stored in memory (150).

[0028] Memory (150) is electrically connected to the processor (140) and stores code executed by the processor (140). More specifically, memory (150) stores a private information detection and masking program executed by the processor (140). Memory (150) stores code that causes the processor (140) to perform the following functions and procedures when executed through the processor (140).

[0029] In memory (150), code is stored that receives a single image (200) containing a reference marker (210) and causes the corner locations (211) for the reference marker (210) to be derived based on the change in pixel intensity of the reference marker (210). The reference marker (210) may be a polygon. For example, if the reference marker (210) is a triangle, three corner locations (211) for the reference marker (210) are derived, and if the reference marker (210) is a square, four corner locations (211) for the reference marker (210) are derived. The change in pixel intensity of the reference marker (210) is derived through the following mathematical formula 1. Because the corners of the reference marker (210) are sharp, the change in pixel intensity around the corners of the reference marker (210) is characterized by being large in the x and y axes. Using these characteristics, the corner position (211) for the reference marker (210) is derived through the following mathematical formula 2.

[0030]

[0031] In mathematical formula 1 represents the amount of change in pixel intensity at a specific pixel location of the reference marker (210), and represents data for a single image (200), and represents the rate of change in pixel brightness in the x-axis direction, and represents the rate of change in pixel brightness in the y-axis direction. It can be derived through a Sobel filter or a Prewitt filter.

[0032]

[0033] In mathematical formula 2 is the magnitude of the change in pixel intensity at a specific pixel location, and represents the sum of the diagonal matrices, and is a constant in the range of 0 to 1.

[0034] In memory (150), code is stored that derives the corner position uncertainty (300) for the corner position (211) based on the change in pixel intensity of the reference marker (210), and derives the position and orientation (400) of the reference marker (210) based on the corner position (211). The position and orientation (400) of the reference marker are derived using a Perspective-n-Point (PnP) algorithm. Perspective-n-Point (PnP) algorithms include the Efficient Perspective-n-Point (EPnP) algorithm, the Uncalibrated Perspective-n-Point (UPnP) algorithm, and the Levenberg-Marquardt algorithm. The position and orientation (400) of the reference marker can be derived by minimizing the value of the following Equation 3 of the Levenberg-Marquardt algorithm.

[0035]

[0036] In mathematical formula 3 , is a corner position (211), and represents a single image (200), represents the corner point of the reference marker (210), and represents the position and orientation of the camera that took a single image (200), and represents the corner position of the reference marker (210) in the world coordinate system, and , represents a value that predicts the corner position of the reference marker (210) in the world coordinate system as a position within a single image (200). For example, if the reference marker (210) is a rectangle, it has four corners, so, is a value for identifying each of the four corners, and can be 1 to 4.

[0037] The corner position uncertainty (300) for the corner position (211) is derived through the following mathematical formula 4. The corner position uncertainty (300) is the reciprocal of the change in pixel intensity of the reference marker (210) derived through mathematical formula 1.

[0038]

[0039] Memory (150) stores code that causes the position and attitude uncertainty (500) of a reference marker (210) to be derived based on the corner position (211) and the corner position uncertainty (300). More specifically, memory (150) stores code that causes the generation of multiple predicted corner points representing the uncertain corner positions of the reference marker (210) based on the corner position (211) and the corner position uncertainty (300). The predicted corner points are derived through at least one of the following mathematical formulas 5 and 6. Multiple predicted corner points can be generated in 2L+1 numbers. By generating only 2L+1 predicted corner points, the amount of computation of the marker attitude uncertainty estimation system can be significantly reduced.

[0040]

[0041]

[0042] In mathematical formulas 5 and 6 is the corner position (211) value, and is the dimension for each corner, and is a scale factor, and is the corner position (211) uncertainty. In Equation 5 is 1,..., L, and in mathematical equation 6 is L+1, ..., 2L.

[0043] Memory (150) stores code that causes the expected position and expected attitude of a reference marker (210) to be derived based on the positions of multiple expected corner points. Memory (150) stores code that causes the average position and attitude estimate for the reference marker (210) to be derived by applying weights to the expected position and expected attitude of the reference marker (210). The weights are derived through Equation 7 below. The average position and attitude estimate for the reference marker (210) is derived through Equation 8 below.

[0044]

[0045] In mathematical formula 7, L is the dimension for each corner of the reference marker (210), and is a scale factor, and can be 1,..., L or L+1, ..., 2L.

[0046]

[0047] In mathematical formula 8 is the expected position and expected orientation of the reference marker (210), and L represents the dimension for each corner of the reference marker (210), It is equal to mathematical formula 7.

[0048] Code is stored that causes the position and attitude uncertainty (500) of the reference marker (210) to be derived based on the average position and attitude estimate of the reference marker (210) and the expected position and expected attitude of the reference marker (210). The position and attitude uncertainty (500) of the reference marker (210) is derived through the following mathematical formula 9.

[0049]

[0050] In mathematical formula 9 is the expected position and expected orientation of the reference marker (210), and L represents the dimension for each corner of the reference marker (210), is an average position and attitude estimate for the reference marker (210).

[0051] In the memory (150), if the position and attitude uncertainty of the reference marker (210) is greater than or equal to a preset value, code is stored that causes the position and attitude of the reference marker (210) to be corrected based on the position and attitude uncertainty of the reference marker (210).

[0052] FIG. 3 is a diagram illustrating an example of generating a corner position and an expected corner point of a reference marker in a single image by a marker pose uncertainty estimation system (100), FIG. 3 (a) is a diagram showing a reference marker (210) in a rectangular shape, FIG. 3 (b) is a diagram showing an enlarged view of a specific corner of the reference marker (210), and FIG. 3 (c) is a diagram showing an example of an expected corner point.

[0053] Referring to FIG. 3, the corner of the reference marker (210) is sharp, so the change in pixel intensity around the corner of the reference marker (210) is large in the x and y axes. The marker pose uncertainty estimation system (100) derives the corner position (211) for the reference marker (210) in a single image (200) by utilizing the characteristic that the change in pixel intensity around the corner of the reference marker (210) is large in the x and y axes. Since the single image (200) containing the reference marker (210) contains the reference marker (210) in which information loss occurs due to the image resolution, noise, lens distortion, and quantization of the camera capturing the single image (200), the pixels around the corner of the reference marker (210) may not be accurately displayed. The marker attitude uncertainty estimation system (100) derives a corner position (211) based on a reference marker (210) in which information loss has occurred, and thus derives a corner position (211) that is different from the actual corner position (212), and can derive the position and attitude of the reference marker (210) based on the corner position (211). Accordingly, the marker attitude uncertainty estimation system (100) can derive the position and attitude uncertainty of the reference marker and increase the reliability of the derived position and attitude of the reference marker (210). The marker attitude uncertainty estimation system (100) generates a plurality of expected corner points (213-1 to 213-3) representing the uncertain corner positions of the reference marker. Multiple predicted corner points (213-1 to 213-3) are sigma points, and the marker attitude uncertainty estimation system (100) derives reference marker position and attitude uncertainty (500) through the multiple predicted corner points (213-1 to 213-3).The marker pose uncertainty estimation system (100) derives reference marker position and pose uncertainty (500) from a single image (200), so it has the advantage that multiple images containing the reference marker are not required when deriving the reference marker position and pose uncertainty (500). In addition, it can significantly reduce the amount of computation compared to artificial intelligence model-based marker recognition technology that requires a large amount of training data. The marker pose uncertainty estimation system (100) generates multiple predicted corner points (213-1 to 213-3) from a single image (200) using a mathematical method and derives the reference marker position and pose uncertainty (500) using them, so it has the advantage of shortening the analysis time compared to a sampling method that collects multiple images or repeatedly detects the position of a specific corner of the reference marker (210) to analyze the distribution of each observation, and uses only a single image (200).

[0054] FIG. 4 is an operation flowchart illustrating a marker pose uncertainty estimation method according to another embodiment of the present invention, and FIG. 5 is a flowchart illustrating detailed steps included in the steps of the marker pose uncertainty estimation method. Hereinafter, the marker pose uncertainty estimation method will be described with reference to FIG. 4 and FIG. 5. Each step of the marker pose uncertainty estimation method described below may be performed by the marker pose uncertainty estimation system (100) described above with reference to FIG. 1 to 3. Accordingly, the content regarding the embodiment of the present invention described above with reference to FIG. 1 to 3 may be applied in the same way to the embodiment described below, and content that overlaps with the description above will be omitted. The steps described below do not necessarily have to be performed in order, the order of the steps can be set in various ways, and the steps may be performed almost simultaneously.

[0055] Referring to FIG. 4, the marker attitude uncertainty estimation method includes a corner position derivation step (S1100), a corner position uncertainty and reference marker attitude derivation step (S1200), and a reference marker attitude uncertainty derivation step (S1300), and further includes a reference marker attitude correction step (S1400).

[0056] The corner position derivation step (S1100) is a step of receiving an image including a reference marker and deriving a corner position for the reference marker based on the change in pixel intensity of the reference marker.

[0057] The corner position uncertainty and reference marker attitude derivation step (S1200) derives the corner position uncertainty for the reference marker corner position based on the change in pixel intensity of the reference marker, and derives the position and attitude of the reference marker based on the corner position.

[0058] The reference marker attitude uncertainty derivation step (S1300) is a step of deriving the position and attitude uncertainty of the reference marker based on the corner position and corner position uncertainty.

[0059] The reference marker attitude correction step (S1400) is a step of correcting the position and attitude of the reference marker based on the position and attitude uncertainty of the reference marker when the position and attitude uncertainty of the reference marker is greater than or equal to a preset value.

[0060] Referring to FIG. 5, the reference marker attitude uncertainty derivation step (S1300) includes an expected corner point generation step (S1310), a reference marker expected attitude derivation step (S1320), a reference marker attitude average estimate derivation step (S1330), and a reference marker position and attitude uncertainty derivation step (S1340).

[0061] The expected corner point generation step (S1310) is a step of generating a plurality of expected corner points representing the uncertain corner location of a reference marker based on the corner location and the corner location uncertainty.

[0062] The reference marker expected attitude derivation step (S1320) is a step of deriving the expected position and expected attitude of a reference marker based on the positions of multiple expected corner points.

[0063] The step of deriving the average estimate of the reference marker posture (S1330) is a step of deriving the average estimate of the position and posture for the reference marker by applying weights to the expected position and expected posture of the reference marker.

[0064] The step of deriving the position and attitude uncertainty of the reference marker (S1340) is a step of deriving the position and attitude uncertainty of the reference marker based on the average estimate of the position and attitude and the expected position and expected attitude of the reference marker.

[0065] The method for estimating marker pose uncertainty of the embodiments of the present invention described above may also be implemented in the form of a recording medium containing computer-executable instructions, such as a program module executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, as well as removable and inseparable media. Additionally, a computer-readable medium may include a computer storage medium. A computer storage medium includes both volatile and non-volatile, removable and inseparable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data.

[0066] A person skilled in the art to which the present invention pertains will understand that, based on the foregoing description, other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention.

Claims

1. A method for estimating marker pose uncertainty performed by at least one processor, wherein a) receiving a single image including a reference marker and deriving a corner position for the reference marker based on the change in pixel intensity of the reference marker; b) deriving a corner position uncertainty for the corner position based on the pixel intensity change amount, and deriving the position and orientation of the reference marker based on the corner position; and c) A method for estimating marker attitude uncertainty, comprising the step of deriving the position and attitude uncertainty of the reference marker based on the corner position and the corner position uncertainty.

2. In Paragraph 1, The above pixel intensity change amount is derived through the following mathematical formula 1, and [Mathematical Formula 1] The above represents the above pixel intensity change amount, and The above is data for the single image above, and The above is the rate of change of pixel brightness in the x-axis direction, and The above A marker pose uncertainty estimation method in which is the rate of change of pixel brightness in the y-axis direction.

3. In Paragraph 2, The corner position for the above reference marker is derived through the following mathematical formula 2, and [Mathematical Formula 2] The above is the magnitude of the change in pixel intensity at a specific pixel location, and The above is the sum of the diagonal matrices, and The above A marker pose uncertainty estimation method in which is a constant within the range of 0 to 1.

4. In Paragraph 2, The uncertainty regarding the above corner position is derived through the following mathematical formula 4, and [Mathematical Formula 4] The above is the change in pixel intensity A marker pose uncertainty estimation method that is the inverse of.

5. In Paragraph 1, The above step c) is, A step of generating a plurality of expected corner points representing the uncertain corner position of the reference marker based on the corner position and the corner position uncertainty; A step of deriving the expected position and expected attitude of the reference marker based on the positions of the plurality of expected corner points; A step of deriving average position and attitude estimates for the reference marker by applying weights to the expected position and expected attitude of the reference marker; and A method for estimating marker attitude uncertainty, comprising the step of deriving the position and attitude uncertainty of the reference marker based on the above-mentioned average position and attitude estimates and the above-mentioned expected position and expected attitude of the reference marker.

6. In Paragraph 5, The above-mentioned expected corner point is derived through at least one of the following mathematical formulas 5 and 6, and [Mathematical Formula 5] [Mathematical Formula 6] In the above mathematical formulas 5 and 6 The above is the value for the above corner position, and The above is the dimension for each corner of the above reference marker, and The above is a scale factor, and The above is the above-mentioned corner position uncertainty, and In the above mathematical formula 5 is 1,..., L and, In the above mathematical formula 6 A marker pose uncertainty estimation method in which L+1, ..., 2L.

7. In Paragraph 6, The above weights are derived through the following mathematical formula 7, and [Mathematical Formula 7] The above L is the dimension for each corner of the above reference marker, and The above is a scale factor, and The above A marker pose uncertainty estimation method in which is 1,..., L or L+1, ..., 2L.

8. In Paragraph 5, The average position and attitude estimates for the above reference marker are derived through the following mathematical formula 8, and [Mathematical Formula 8] The above is the expected position and expected attitude of the above reference marker, and The above A marker pose uncertainty estimation method in which is the dimension for each corner of the above-mentioned reference marker.

9. In Paragraph 5, The position and attitude uncertainty of the above reference marker is derived through the following mathematical formula 9, and [Mathematical Formula 9] The above is the expected position and expected attitude of the above reference marker, and The above is the dimension for each corner of the above reference marker, and The above A marker attitude uncertainty estimation method in which is an average estimate of the position and attitude for the above reference marker.

10. In Paragraph 1, d) A method for estimating marker attitude uncertainty, further comprising the step of correcting the position and attitude of the reference marker based on the position and attitude uncertainty of the reference marker if the position and attitude uncertainty of the reference marker is greater than or equal to a preset value.

11. At least one processor; and It includes a memory electrically connected to the above processor and storing codes executed in the above processor, The above memory is, the processor, A single image including a reference marker is received, and a corner position for the reference marker is derived based on the change in pixel intensity of the reference marker. Based on the above pixel intensity change amount, derive the corner position uncertainty for the above corner position, derive the position and orientation of the above reference marker based on the above corner position, and, A marker attitude uncertainty estimation system that stores a code causing the position and attitude uncertainty of the reference marker to be derived based on the corner position and the corner position uncertainty.

Images