Mark detection system, positioning system, mark detection method, mark detection program, and connected component detection system

Abstract

This mark detection system (100) detects the positions of a plurality of geometric marks included in an image and overlapping each other. The mark detection system (100) comprises: an input unit (10) having an image input unit (11) for inputting an input image, and a contraction parameter input unit (12) for inputting a parameter relating to contraction processing; a signal processing unit (20) for repeatedly executing contraction processing on the input image using the contraction parameter, and separating and detecting a plurality of geometric marks on the basis of the execution result; and an output unit (40) for outputting the processing result of the signal processing unit (20).

Description

Mark detection system, positioning system, mark detection method, mark detection program, and linked component detection system

[0001] This disclosure relates to a technology for detecting the location of a mark placed on an object.

[0002] In the manufacturing of semiconductor products and display devices, a process is carried out to position (align) two objects, such as a substrate and a glass substrate, relative to each other. In the alignment process, the positions of marks placed on the two objects are controlled so that they are in a predetermined relative position, thereby aligning the positions of the two objects. After this positioning process, for example, a process of bonding the two objects together, or a process of using the glass material as a photomask to transfer a circuit onto a silicon object is carried out.

[0003] Patent Document 1 discloses a technique for detecting surface defects of an object using image processing. In this technique, small images are compressed by performing image dilation / collapse processing, and the difference is detected as a defect by comparing it with the input image.

[0004] Patent document 2 discloses a technique for detecting the position of a mark on a substrate. This technique performs pattern matching using an image of a reference mark as a template.

[0005] Japanese Patent Publication No. 6812118, Japanese Unexamined Patent Publication No. 2022-21438

[0006] In the alignment process, multiple marks placed on an object may overlap in the captured image. In such cases, it is necessary to perform high-speed separation detection of the marks in the captured image. However, the aforementioned Patent Documents 1 and 2 do not address this problem.

[0007] In view of the foregoing, this disclosure aims to enable high-speed separation detection of marks when marks attached to an object overlap in the captured image.

[0008] A mark detection system according to one aspect of the present disclosure detects the positions of a plurality of overlapping geometric marks contained in an input image and comprises an input unit having an image input unit for inputting the input image and a shrinkage parameter input unit for inputting parameters related to shrinkage processing; a signal processing unit that repeatedly performs the shrinkage processing on the input image using the parameters and separates and detects the plurality of geometric marks based on the results of this execution; and an output unit that outputs the processing results of the signal processing unit.

[0009] This disclosure enables high-speed detection and separation of marks when marks attached to an object overlap in the captured image.

[0010] Examples of the configuration of a positioning system utilizing mark detection according to the embodiment (a), (b) are examples of marks attached to an object, etc., (c) is an example of a captured image (a), (b) is an example of a binarized image containing multiple marks Conceptual diagram showing the processing flow of mark detection according to the embodiment Example of the configuration of a mark detection system according to Embodiment 1 Basic flow of the mark detection method according to Embodiment 1 Flow of the vanishing point generation process in the basic flow of Diagram 6 Flow of the shrinkage process in the vanishing point generation process in Diagram 7 Flow of the vanishing point classification process in the basic flow of Diagram 6 Flow of the calculation process for each mark position in the basic flow of Diagram 6 Flow of the success / failure determination process in the basic flow of Diagram 6 (a), (b) are examples of user interface screens Basic flow of the mark detection method according to Embodiment 2 Basic Flowchart 13 shows the first example of the success / failure determination process in this flowchart. Flowchart 13 shows the second example of the success / failure determination process in the basic flowchart. configuration of the positioning system according to Embodiment 3. Basic flowchart 3 shows the control according to Embodiment 3. Basic flowchart 4 shows the control according to Embodiment 4. Configuration example of a processing machine that executes the basic flowchart 18 shows the operation example of a processing machine in Figure 19. Timing chart showing an example of a processing machine that executes the basic flowchart 19 shows an example of a processing machine that executes the processing machine in Figure 19. Other examples of user interface screens Other examples of user interface screens (a) and (b) show other examples of marks attached to objects, etc. Example of evaluation results of this disclosure Example of evaluation results of this disclosure Example of applying the method of this disclosure to distance images Example of applying the method of this disclosure to tactile sensing data Example of shrinkage processing for network data The process of network data shrinking

[0011] (Summary) A mark detection system according to an aspect of the present disclosure detects the positions of a plurality of geometric shape marks overlapping each other included in an input image, and includes an image input unit that inputs the input image, and a shrinkage parameter input unit that inputs parameters related to shrinkage processing. An input unit, a signal processing unit that repeatedly executes the shrinkage processing on the input image using the parameters and separates and detects the plurality of geometric shape marks based on the execution result, and an output unit that outputs the processing result of the signal processing unit.

[0012] According to this configuration, the mark detection system inputs, by the input unit, an input image including a plurality of geometric shape marks overlapping each other and parameters related to shrinkage processing. The signal processing unit repeatedly executes the shrinkage processing on the input image using the parameters and separates and detects a plurality of geometric shape marks based on the execution result. The output unit outputs the processing result of the signal processing unit. Thereby, the separation detection of a plurality of geometric shape marks overlapping in the image can be executed at high speed.

[0013] In the mark detection system according to the above aspect, the signal processing unit may extract the position coordinates of vanishing points that disappeared in the process of repeating the shrinkage processing, classify the vanishing points for which the position coordinates have been extracted based on the number of times of shrinkage processing until disappearance, and obtain the position coordinates of the geometric shape marks from the position coordinates of the vanishing points of each class.

[0014] Thereby, the signal processing unit extracts the position coordinates of vanishing points that disappeared in the process of repeating the shrinkage processing. The vanishing points for which the position coordinates have been extracted are classified based on the number of times of shrinkage processing until disappearance. Then, the position coordinates of the geometric shape marks can be obtained from the position coordinates of the vanishing points of each class. Therefore, the positions of a plurality of geometric shape marks can be detected at high speed.

[0015] Further, the parameter may include a threshold related to the number of times of shrinkage processing until disappearance for classifying vanishing points.

[0016] Thereby, the threshold for classifying vanishing points can be adjusted.

[0017] In the mark detection system according to the above aspect, the parameter may include the number of geometric shape marks included in the input image.

[0018] In the mark detection system according to the above aspect, the parameter may include the maximum number of repetitions of the shrinking process.

[0019] In the mark detection system according to the above aspect, the input unit and the output unit include a display having a user interface screen, and the user interface screen includes an area for displaying the input image, an area for setting the parameter, and an area for displaying the result of the separation detection process of the plurality of geometric shape marks.

[0020] Thereby, the user can check the input image on the user interface screen, set the parameters related to the shrinking process, and also check the result of the separation detection process of the geometric shape marks.

[0021] In the mark detection system according to the above aspect, a processing result determination unit for determining the processing result of the signal processing unit is provided, and when the processing result determination unit determines that reprocessing is necessary, the parameter is changed so that the signal processing unit separates and detects the plurality of geometric shape marks again.

[0022] Thereby, when it is determined that reprocessing is necessary for the processing result of the signal processing unit, the parameter is changed and the separation detection of the plurality of geometric shape marks is performed again, so that the accuracy of the separation detection process can be improved.

[0023] Furthermore, in the mark detection system according to the above aspect, the maximum number of executions of the reprocessing of the separation detection may be configured to be settable.

[0024] A positioning system according to an aspect of the present disclosure positions a first object marked with a first geometric mark and a second object marked with a second geometric mark, and comprises a mark detection system that separates and detects the first and second geometric marks from an image of the first and second geometric marks, a moving unit that moves the position of at least one of the first and second objects, and a control unit that controls the amount of movement by the moving unit based on the output of the mark detection system, wherein the mark detection system comprises an input unit having an image input unit for inputting the image and a shrinkage parameter input unit for inputting parameters related to shrinkage processing, a signal processing unit that repeatedly performs the shrinkage processing on the image using the parameters and separates and detects the first and second geometric marks based on the results of this execution, and an output unit that outputs the processing results of the signal processing unit.

[0025] In this configuration, the positioning system positions a first object marked with a first geometric mark and a second object marked with a second geometric mark. The mark detection system separates and detects the first and second geometric marks from an image capturing the first and second geometric marks. Based on the output of the mark detection system, the system controls the amount of movement to move at least one of the first and second objects. The mark detection system receives the input image and parameters related to the shrinkage process via the input unit. The signal processing unit repeatedly performs the shrinkage process on the input image using the parameters, and based on the results of this process, separates and detects multiple geometric marks. The output unit outputs the processing results from the signal processing unit. This enables high-speed separation and detection of overlapping first and second geometric marks in the image. Therefore, even if the first and second geometric marks overlap in the image, the object can be positioned accurately.

[0026] In the positioning system according to the above embodiment, the mark detection system may determine whether the first and second geometric shape marks overlap in the image, and if it determines that they do not overlap, it may detect the position coordinates of the first and second geometric shape marks, respectively, without using the shrinkage process.

[0027] As a result, when the mark detection system determines that the first and second geometric shape marks do not overlap in the image, it detects the position coordinates of the first and second geometric shape marks respectively without using a shrinking process.

[0028] A mark detection method according to an aspect of the present disclosure detects the positions of multiple overlapping geometric marks included in an input image, and comprises the steps of: repeatedly performing a shrinking process on the input image; extracting the position coordinates of vanishing points that disappear during the process of repeating the shrinking process; classifying the vanishing points whose position coordinates have been extracted based on the number of shrinking processes performed until they disappear; and determining the position coordinates of geometric marks from the position coordinates of the vanishing points in each class.

[0029] In this configuration, for input images containing multiple overlapping geometric marks, a shrinking process is repeatedly performed, and the position coordinates of vanishing points that disappear during the shrinking process are extracted. The extracted vanishing points are then classified based on the number of shrinking processes performed until they disappear. The position coordinates of the geometric marks are then determined from the position coordinates of the vanishing points in each class. This allows for high-speed detection of the position of each geometric mark, even when multiple geometric marks overlap in an image.

[0030] In the mark detection method according to the above embodiment, the number of the plurality of geometric marks may be set as the number of classes for classifying the vanishing points.

[0031] This allows for more accurate separation and detection of multiple geometric marks.

[0032] In the mark detection method according to the above embodiment, the number of geometric marks, a threshold for classifying vanishing points, and the maximum number of repetitions of the shrinkage process may be input as preprocessing.

[0033] This allows for adjustment of the accuracy of mark separation detection.

[0034] This disclosure includes a program for causing a computer to perform the mark detection method according to the above embodiment.

[0035] A linked component detection system according to an aspect of the present disclosure detects the positions of a plurality of mutually overlapping linked components included in input data, and comprises: an input unit having a data input unit for inputting the input data and a shrinkage parameter input unit for inputting parameters related to shrinkage processing; a signal processing unit that repeatedly performs the shrinkage processing on the input data using the parameters and separates and detects the plurality of linked components based on the results of this execution; and an output unit that outputs the processing results of the signal processing unit.

[0036] In this configuration, the connected component detection system receives input data containing multiple overlapping connected components and parameters related to the condensation process via the input unit. The signal processing unit repeatedly performs the condensation process on the input data using the parameters, and based on the results of this process, separates and detects the multiple connected components. The output unit outputs the processing results of the signal processing unit. This enables high-speed separation and detection of multiple overlapping connected components in the data.

[0037] (Embodiments) Hereinafter, embodiments will be described in detail with reference to the drawings.

[0038] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components.

[0039] Figure 1 shows an example of a positioning system configuration, which is an example of a case using the mark detection system according to the embodiment. In Figure 1, the position of the object 2 (e.g., a substrate) placed on the stage is movable by the motor 3. Based on the image captured by the camera 4, the FPGA (Field Programmable Gate Array) 5 detects the positional relationship between the object 2 and the glass material 6 as a second object. The PLC (Programmable Logic Controller) 7 moves the position of the object 2 by controlling the motor 3, thereby positioning the object 2 and the glass material 6 relative to each other.

[0040] As shown in Figure 2(a), object 2 has a circle as a geometric mark (hereinafter referred to as "mark" as appropriate). As shown in Figure 2(b), glass material 6 has a ring (double circle) as a mark. As shown in Figure 2(c), positioning (alignment) is performed so that in the image captured by camera 4, the circle mark of object 2 is inside the ring mark of glass material 6. Note that the state of the marks targeted by the alignment is not limited to the state shown in Figure 2(c); positioning is performed so that the positions of the circle of object 2 and the ring of glass material 6 are in a predetermined relative position. Note that the mark here does not necessarily have to be a circle or a ring, and its size is not limited.

[0041] Figure 3 shows an example of a binarized image containing two marks. In the image in Figure 3(a), the two marks are separated from each other, making it easy to detect the position of each mark. On the other hand, in the image in Figure 3(b), the two marks overlap in the image. In this state, it is difficult to detect the position of each mark, so it is necessary to separate the marks before detecting their positions. Moreover, this mark separation and detection process needs to be performed at high speed.

[0042] Figure 4 is a conceptual diagram illustrating the algorithm of a mark detection method according to an embodiment. In this algorithm, overlapping marks in an image are separated using a shrinkage process, taking advantage of the difference in the "thickness" of each mark. The shrinkage process is a process that reduces the white pixel area in a binarized image and is performed for purposes such as noise reduction.

[0043] Currently, the binarized image contains two overlapping marks as white pixels. This binarized image is repeatedly subjected to a shrinking process. When a white pixel disappears during the repeated shrinking process, its coordinate position is identified as a "vanishing point" (vanishing point search). In addition, pixels that remain as white pixels even after the maximum number of shrinking operations are performed are also identified as "vanishing points".

[0044] Then, for each vanishing point, the number of repetitions of the shrinkage process required for it to disappear, i.e., the "number of shrinkages," is determined. For example, when a white pixel at coordinates (i,j) disappears after the N (N is a natural number) shrinkage process, the coordinates of the vanishing point are (i,j), and the number of shrinkages is N.

[0045] The identified vanishing points are classified into multiple classes based on the number of contractions (contraction count classification). In this case, the number of classes can be set to the number of overlapping marks, for example, if the number of overlapping marks is known. For example, if it is known that two marks overlap, the vanishing points are classified into two classes.

[0046] Then, in each class, the position of each mark is calculated based on the position coordinates of the vanishing point.

[0047] This algorithm significantly reduces the computational complexity compared to similarity calculations in conventional, basic template matching methods, enabling high-speed detection of mark separation.

[0048] (Example 1) Figure 5 is a block diagram showing the configuration of the mark detection system according to Example 1. The mark detection system 100 shown in Figure 5 comprises an input unit 10, a signal processing unit 20, a processing result determination unit 30, and an output unit 40.

[0049] The input unit 10 includes an image input unit 11 for inputting an image of the marks, and a shrinkage parameter input unit 12 for inputting parameters related to the shrinkage process. The shrinkage parameters include, for example, the number of marks in the image and an initial value for the maximum number of iterations of the shrinkage process. The input unit 10 receives input images from, for example, storage (not shown) via an interface such as USB, Gigabyte, or CameraLink. Alternatively, the input unit 10 may receive input images from a camera via an interface.

[0050] The signal processing unit 20 performs a process to separate and detect marks contained in the image, and comprises a separation processing unit 21 and a detection processing unit 22. The separation processing unit 21 repeatedly performs a shrinking process on the image to identify vanishing points and classify each vanishing point into multiple classes. The detection processing unit 22 uses the coordinate positions of the vanishing points in each class to detect the position of each mark.

[0051] The processing result determination unit 30 determines the processing result of the signal processing unit 20, and specifically, for example, estimates the detection accuracy of each mark detected by the signal processing unit 20. The mark detection system 100 has a closed loop that allows the signal processing unit 20 to perform mark separation detection again by changing the contraction parameter when the processing result determination unit 30 determines that the detection accuracy is low and reprocessing is necessary.

[0052] The output unit 40 outputs the position information of the separated marks. The output unit 40 transmits the mark position information to the on-screen UI (User Interface) via an interface such as Ethernet.

[0053] The signal processing unit 20 and the processing result determination unit 30 are implemented, for example, by a processor, CPU (Central Processing Unit), FPGA, ASIC (Application Specific Integrated Circuit), etc., capable of executing a program stored in memory.

[0054] The input unit 10 and output unit 40 may, for example, be equipped with a display having a UI screen. However, the input unit 10 and output unit 40 do not necessarily have a UI screen. Examples of input units 10 without a UI screen include configurations with a UI such as switches or levers, or interfaces for input from external devices. Similarly, examples of output units 40 without a UI screen include interfaces for outputting to external devices such as robots or industrial equipment.

[0055] Figure 6 shows the basic flow of the mark detection method according to Embodiment 1. In Figure 6, when the mark detection system 100 starts operating, an image of an object with a mark is input to the mark detection system 100 and is binarized by the image input unit 11 (S11). Alternatively, an already binarized image may be input to the mark detection system 100.

[0056] Parameters for performing mark detection are set by the shrink parameter input unit 12 (S12). These parameters include, for example, the number of marks to detect and the maximum number of shrink process repetitions. Additionally, a threshold value for the number of shrink processes until disappearance is set as a parameter for classifying vanishing points into different classes. These parameters are set by the user, for example, via a screen-based UI.

[0057] In the vanishing point generation process S20, vanishing points are extracted by repeatedly applying a shrinking process to the input image. In the vanishing point classification process S30, the vanishing points extracted in the vanishing point generation process S20 are classified into classes based on the number of shrinking processes required for their disappearance. In the mark position calculation process S40, the position of each mark is calculated from the coordinates of the vanishing points included in each class. In the success / failure determination process S50, it is determined whether or not reprocessing is necessary for the mark position detection. Details of the vanishing point generation process S20, the vanishing point classification process S30, the mark position calculation process S40, and the success / failure determination process S50 will be described later.

[0058] If reprocessing is not necessary (no in S13), the positions of each detected mark are output (S14). On the other hand, if reprocessing is necessary (yes in S13), the process returns to step S12, the parameters are reset, and the process is performed again.

[0059] Figure 7 shows the flow of the vanishing point generation process S20 in the basic flow of Figure 6.

[0060] In Figure 7, first, buffer image I Ｂ And, image I of the number of contractions ｉ Prepare (S21). Buffer image I Ｂ This is an image that stores the pixel values ​​during the process of repeating the shrinkage process S60, and is a shrinkage count image I. ｉ This is an image that stores the number of times the shrinking process is repeated. Also, the shrinking count iterator n is initialized to 0 (S22).

[0061] Then, the condensation process S60 is executed. Details of the condensation process S60 will be described later. After the condensation process S60 is executed, the buffer image I Ｂ If any pixel value within indicates 1 (yes in S23) and n does not exceed the maximum number of shrinkage counts N (yes in S24), then n is incremented (S25) and the shrinkage process S60 is executed again. Meanwhile, buffer image I Ｂ If all the pixel values ​​within are 0 (no in S23), or if n has reached N (no in S24), proceed to step S26.

[0062] In step S26, the number of contractions image Iｉ Record the local maximum value in ｉ together with its coordinates in the vanishing point data sequence V.

[0063] FIG. 8 shows the flow of the shrinking process S60 in the vanishing point generation process S20 of FIG. 7. FIG. 8 shows the iteration in the case of n = n'.

[0064] Set the target pixel coordinates (u, v) (S61). Then, for the input image I ｓ check whether one or more of the pixel values of the eight pixels (I ｓ (u - 1, v - 1), I ｓ (u, v - 1), I ｓ (u + 1, v - 1), I ｓ (u - 1, v), I ｓ (u + 1, v), I ｓ (u - 1, v + 1), I ｓ (u, v + 1), I ｓ (u + 1, v + 1)) near the target pixel coordinates (u, v) in ｓ indicate 0 (S62).

[0065] When one or more of the pixel values indicate 0 (yes in S62), input 0 to the value of the target pixel coordinates (u, v) in the buffer image I Ｂ . On the other hand, when none of the pixel values indicate 0, in other words, when all of the pixel values indicate 1 (no in S62), input 1 to the value of the target pixel coordinates (u, v) in the buffer image I Ｂ and input n' to the value of the target pixel coordinates (u, v) in the shrinkage count image I ｉ (S65).

[0066] When the target pixel is the last pixel (yes in S66), end the shrinking process S60. Otherwise (no in S66), set the next target pixel coordinates (u, v) (S67) and return to step S62. The setting of the next target pixel may be, for example, to select the pixel on the right side of the current pixel if there is a pixel on the right side, or to select the leftmost pixel in the next row of the current pixel if there is no pixel on the right side.

[0067] Figure 9 shows the flow of the vanishing point classification process S30 in the basic flow of Figure 6. In Figure 9, the vanishing points extracted in the vanishing point generation process S20 are classified into two classes.

[0068] The first data point of the vanishing point data sequence V is designated as the reference data (S31). Then, it is determined whether the local maximum value recorded in the reference data is greater than the threshold t (S32). If the local maximum value is greater than the threshold t (yes in S32), the coordinates of the reference data are set to the data sequence V of vanishing point class 1. １ The data is stored in (S33). On the other hand, when the local maximum value is less than or equal to the threshold t (no in S32), the coordinates of the reference data are stored in the data column V of vanishing point class 2. ２ The data is stored in (S34). Then, the reference data is deleted from the vanishing point data column V (S35). These processes are repeated until there is no more data left in the vanishing point data column V (S36).

[0069] Figure 10 shows the flow of the mark position calculation process S40 in the basic flow of Figure 6. In step S41, the data sequence V of vanishing point class 1 １ The average of all coordinates in the data stored in is the center coordinate (u) of Mark 1. １ ,v １ ) is recorded as. In step S42, the data sequence V of vanishing point class 2 ２ The average of all coordinates in the data stored in is the center coordinate (u) of Mark 2. ２ ,v ２ Record it as follows:

[0070] Figure 11 shows the processing flow of the success / failure determination process S50 in the basic flow of Figure 6. In step S51, the data sequence V of vanishing point class 1 １ , and data sequence V of vanishing point class 2 ２ The number of elements is determined to be 1 or greater for each element. If all are 1 or greater (yes in S51), it is determined that reprocessing is not necessary (S52). On the other hand, if at least one of them is 0 (no in S51), it is determined that reprocessing is necessary (S53).

[0071] Figures 12(a) and 12(b) show examples of UI screens. The UI screen in Figure 12 includes an area A1 for displaying the input image, an area A2 for setting the shrinkage parameters, and an area A3 for displaying the results of the mark separation detection process. Figure 12(a) shows the case when the mark detection process is successful, and Figure 12(b) shows the case when the mark detection process fails.

[0072] In area A1, the input image is displayed. In area A2, there are fields to input the number of marks to be detected and the maximum number of repetitions for contraction. Also in area A2, there is a button to set whether or not to display the estimated position (coordinates of the detected marks) in the image. In area A3, the number of detected marks and the detection position of each mark are displayed. Also in area A3, there is a display B1 that evaluates the processing result. For example, the evaluation shown by this display B1 is based on whether or not the number of detected marks matches the set target number of marks.

[0073] In the UI screen shown in Figure 12, the user can view the input image and set parameters related to the shrinking process. They can also view the results of the geometric shape mark separation detection process.

[0074] In this embodiment, the mark detection system 100 receives a plurality of overlapping geometric marks and parameters related to the shrinking process as input from the input unit 10. The signal processing unit 20 repeatedly performs the shrinking process on the input image using the parameters, and based on the results of this process, separates and detects the plurality of geometric marks. The output unit 40 outputs the processing results of the signal processing unit 20. This enables high-speed separation and detection of multiple overlapping geometric marks in an image.

[0075] Furthermore, the mark detection system 100 includes a processing result determination unit 30. When the processing result determination unit 30 determines that reprocessing is necessary for the processing result of the signal processing unit 20, it changes the parameters, and the signal processing unit 20 performs separation detection of multiple geometric marks again. This improves the accuracy of the mark separation detection process. Note that the processing result determination unit 30 may be omitted from the mark detection system 100.

[0076] (Example 2) In Example 2, the configuration of the mark detection system is the same as in Example 1. That is, as shown in Figure 5, it comprises an input unit 10, a signal processing unit 20, a processing result determination unit 30, and an output unit 40.

[0077] Figure 13 shows the basic flow of the mark detection method according to Example 2. The flow in Figure 13 is basically the same as in Figure 6 in Example 1. However, in this example, an execution count iterator m is introduced, and rules are defined regarding the number of reprocessing executions. In step S15, m is initialized. The content of the success / failure determination process S50A differs from that of Example 1.

[0078] Figure 14 shows a first example of the success / failure determination process S50A in the basic flow of Figure 13. Figure 14 shows the process when m = m'.

[0079] When m' is less than the maximum number of executions M (yes in S54), in step S51, the data sequence V of vanishing point class 1 １ , and data sequence V of vanishing point class 2 ２ The number of elements is determined to be 1 or greater for each element. If all are 1 or greater (yes in S51), the coordinates (u) of the detected mark 1 are determined. １ ,v １ ) to the coordinate buffer column C of Mark 1 １ Add to the detected mark 2 coordinates (u ２ ,v ２ ) to the coordinate buffer column C of Mark 2 ２ It is added to (S55) and determined that no reprocessing is necessary (S52). Meanwhile, the data sequence V of vanishing point class 1 １ , and data sequence V of vanishing point class 2 ２ If at least one of the elements is 0 (no in S51), it is determined that reprocessing is necessary (S53), and m is incremented (S56).

[0080] Figure 15 shows a second example of the success / failure determination process S50A in the basic flow of Figure 13. In Figure 15, the process for the case where m = m' is also shown.

[0081] When m' is less than the maximum number of executions M (yes in S54), in step S51, the data sequence V of vanishing point class 1 １ , and data sequence V of vanishing point class 2 ２ The number of elements is determined to be 1 or greater for each element. If all are 1 or greater (yes in S51), the coordinates (u) of the detected mark 1 are determined. １ ,v １ ) to the coordinate buffer column C of Mark 1 １ Add to the detected mark 2 coordinates (u ２ ,v ２ ) to the coordinate buffer column C of Mark 2 ２ Add to (S55).

[0082] In step S57, coordinate buffer column C １ , C ２ In each of these, the distance Δd is between the latest coordinate and the mean of the coordinates in the array. １ , Δd ２ Calculate Δd. １ The threshold α １ Smaller than Δd ２ The threshold α ２ When it is smaller than (yes in S58), it is determined that the process has converged and that reprocessing is not necessary (S52). On the other hand, Δd １ The threshold α １ That is the case, or Δd ２ The threshold α ２ If the result is greater than or equal to (no in S58), m is incremented (S56), and it is determined that reprocessing is necessary (S53).

[0083] On the other hand, in step S51, data sequence V １ , V ２ If at least one of the elements is 0 (no in S51), m is incremented (S56) and it is determined that reprocessing is necessary (S53).

[0084] (Example 3) Figure 16 shows the configuration of the positioning system according to Example 3. The positioning system 300 in Figure 16 positions a first object marked with a first mark and a second object marked with a second mark. The positioning system 300 includes the mark detection system 100 according to Example 1 or 2, and further includes an imaging unit 310, a control unit 320, and a moving unit 330. The moving unit 330 has the function of moving the position of at least one of the first and second objects to be positioned. The control unit 320 controls the amount of movement by the moving unit 330 based on the output of the mark detection system 100. In the configuration example of Figure 1, for example, the camera 4 corresponds to the imaging unit 310, the FPGA 5 corresponds to the mark detection system 100, the PLC 7 corresponds to the control unit 320, and the motor 3 corresponds to the moving unit 330. In the configuration shown in Figure 1, the motor 3 moves the object 2 on the stage, but the configuration may also be such that the glass material 6 is moved, or both the object 2 and the glass material 6 are moved.

[0085] Figure 17 shows the basic flow including control according to Embodiment 3. The flow in Figure 17 includes, in addition to the basic flow of Embodiment 1 or 2, a duplication determination step S71 which determines whether or not the marks overlap in the image captured by the imaging unit 310.

[0086] In the duplicate detection step S71, it is determined whether or not the marks are duplicated by, for example, performing blob detection. In blob detection, for example, a set of adjacent foreground images of a predetermined size is detected as a single blob. If the number of target marks is known and the number of detected blobs is less than this number, it can be determined that the marks are duplicates.

[0087] If it is determined that there is a duplicate (yes in S72), the mark detection method according to Example 1 or Example 2 is executed. On the other hand, if it is determined that there is no duplicate (no in S72), the non-duplicate detection step S73 is performed. This step S73 is achieved, for example, by calculating the position of each blob detected in the duplicate determination step S71.

[0088] When the detection position of the mark is output, the control unit 320 calculates a control amount to adjust the position of the object (S74), and outputs the calculated control amount information to the movement unit 330 (S74). As a result, for example, in the configuration shown in Figure 1, the position of the object is adjusted.

[0089] According to this embodiment, the positioning system 300 includes the mark detection system 100 according to Embodiment 1 or 2. The mark detection system 100 can quickly separate and detect overlapping marks in an image. Based on the output of the mark detection system 100, the amount of movement to move the object's position is controlled. Therefore, even if marks attached to an object in an image overlap, the object can be positioned accurately.

[0090] (Example 4) Figure 18 shows the basic flow including control according to Example 4. In the flow shown in Figure 17 in Example 3, the determination of whether or not the marks overlap was performed at the beginning of the process. In contrast, in the flow shown in Figure 18, the determination of whether or not the marks overlap is performed in the latter half of the process. Also, in the flow shown in Figure 18, the relative position information between the marks is output. The control amount of the control device can be determined by this relative position information.

[0091] Figure 19 shows an example of a processor configuration for executing the flow shown in Figure 18. The processor 400 shown in Figure 19 includes an FPGA 410. The FPGA 410 has a binarization processing unit 411, a separation processing unit 420, and a blob detection processing unit 412. The separation processing unit 420 includes a stenosis processing unit 430 and an image memory 421. The stenosis processing unit 430 has multiple stenosis units 431 that perform one stenosis at a time, all implemented in parallel, and each stenosis unit 431 is provided with its own memory unit 432. This is a more suitable configuration for high-speed execution.

[0092] In Figure 18, the region enclosed by the bars is executed in parallel, for example. The processes in S12, S20, and S30 are as previously described. The blob detection process S81 is the same as the non-overlap detection process S73 in the flow of Figure 17 shown in Example 3, and for example, it detects a set of adjacent foreground images of a predetermined size as a single blob. The parallel execution here is performed, for example, by a processing machine equipped with an FPGA as shown in Figure 19. However, it is not limited to parallel execution. Parallel execution may also be performed by a processing machine other than an FPGA.

[0093] In each mark position calculation process S40A, the mark positions are calculated based on the process in S40, which has already been explained, and the blobs detected in S81. Then, it is determined whether or not the marks overlap (S82). If it is determined that they overlap (YES in S82), the relative positions between the marks are calculated using the mark positions calculated from the vanishing points (S83). On the other hand, if it is determined that the marks do not overlap (NO in S82), the relative positions between the marks are calculated using the mark positions calculated using the blobs (S84). Finally, the calculated relative position information is output (S85).

[0094] In the configuration shown in Figure 19, when the shrinking unit 431 performs a shrinking process, the corresponding memory unit 432 stores the number of shrinking cycles and the image after the shrinking process. The stored data is then used for the next shrinking process by the shrinking unit 431.

[0095] In FPGA 410, image data is typically read one line at a time. For a single scaling operation, it is usually sufficient if only the eight neighboring pixels of a given pixel are available. Therefore, the scaling operation can be performed if only three lines, including the upper and lower lines, are input, and for this reason, only two lines of line memory connected to the scaling operation are sufficient. In other words, the data for the two lines stored in line memory and the data for the one line directly input from the binarization processing unit 411 total three lines.

[0096] Figure 20 is a timing chart showing an example of the operation of the processing machine in Figure 19. As mentioned above, it requires only two lines of line memory, resulting in low latency. The first line of the image (typically the top line) has no lines above it, so the scaling process can be performed using only the first and second lines combined. In other words, as soon as the second line of the input image begins to be read, the first scaling can begin immediately. After that, the scaling process for the next line can be performed each time a line is read. Once the result of the scaling process for the second line is obtained, the second scaling of the first line can be performed immediately.

[0097] (Other Examples) Figure 21 shows another example of the UI screen. In Figure 21, in the parameter setting area A2, the threshold value for the number of contractions used to separate the vanishing point can be adjusted using a trackbar. A histogram C1 of the number of contractions required for vanishing is also displayed, making it easier to set the threshold value.

[0098] Figure 22 shows another example of the UI screen. This example screen assumes that the mark detection system 100 has a function to evaluate the processing results internally and perform detection processing for multiple iterations while automatically adjusting the number of contractions. In the parameter setting area A2, the upper limit of the iterations can be set. In addition, the threshold for separating the vanishing points is set in a table. Furthermore, in the processing result area A3, the number of iterations repeated until mark separation is successful is displayed. Success in mark separation is determined, for example, when the detection position does not change significantly even when the number of contractions is changed. The convergence value of the number of contractions is also displayed. By using this value in subsequent executions, it is expected that processing time will be shortened and accuracy will be improved.

[0099] In the embodiments described above, examples were given in which a circle was placed on the object and a ring was placed on the glass material as geometric marks. However, the marks placed on the object are not limited to circles or rings; for example, as shown in Figure 23(a), four squares arranged in a row may be used, or as shown in Figure 23(b), a cross may be used. Furthermore, the objects to which the marks are placed are not limited to substrates or glass materials. Also, the number of marks included in the image is not limited to two.

[0100] (Evaluation Results) Figure 24 shows an example of the evaluation results of this disclosure. A simulation was created of a scene in which a circle and a ring mark overlap, as shown in Figure 24(a). The circle has a radius of 50 pixels, and the ring has an inner radius of 70 pixels. The X-direction error and Y-direction error are calculated as shown in Figure 24(b). As shown in Figure 24(c), the thinner the line width of the ring, the better the results (the accuracy improves). Also, at least in the range shown in Figure 24(c), the vanishing points can be classified into two when the circle and ring overlap.

[0101] Figure 25 shows another example of the evaluation results of this disclosure. A real-world scene of a circle and a ring overlapping was prepared. A silicon (Si) substrate and a glass substrate were prepared, and a circle was printed on the silicon substrate and a ring was printed on the glass substrate. Gold (Au) was used as the printing material. The circle has a radius of 0.5 mm, and the ring has an inner radius of 0.75 mm and a line width of 0.05 mm. The camera was set to capture approximately 0.01 mm per pixel.

[0102] Figure 25 is a graph showing the error values ​​as a histogram, calculated by taking 1000 frames of images for a single scene and calculating the error from the true position. Figure 25(a) is a graph for scenes where the circle and ring overlap (overlapping scenes), and Figure 25(b) is a graph for scenes where the circle and ring do not overlap (non-overlapping scenes). As shown in Figure 25(a), in overlapping scenes, the variability can be suppressed to a degree smaller than the difference between the circle radius and the inner radius of the ring. As shown in Figure 25(b), in non-overlapping scenes, the non-overlapping nature can be well identified and the variability can be suppressed.

[0103] Figure 26 shows an example of the evaluation results of this disclosure. Figure 26 shows the timing from the start of image data input to the end of position calculation. As can be seen from Figure 26, this disclosure can achieve a processing time of about 1 ms.

[0104] <Other Embodiments Regarding Image Processing> (Application to Defect Detection) In the embodiments described above, a method for detecting the position of a mark from an image was explained. However, the objects to be detected from an image using this method are not limited to marks. For example, in the manufacturing or assembly processes of semiconductors and electronic components, this method can be used to detect defects such as dust or scratches attached to the manufactured object from an image. Specifically, for example, in an image-based defect detection application, if an image is detected at an unexpected location after detection using this method, that image can be detected as a defect. Alternatively, images other than defects can be excluded from the input image by image subtraction, etc., and then this method can be applied to detect defects. In this way, this method can be applied to defect detection.

[0105] (Application to depth images) Figure 27 is an illustrative diagram showing an example of applying the above method to depth images. As shown in Figure 27, in generating depth images, the measurement range is divided into fixed intervals, and interval image data lm1 to lm3 are obtained, which represent in binary whether or not an object is present in each interval. Since each interval image lm1 to lm3 is a binary image, the positions of overlapping objects can be separated and detected by using the above method.

[0106] There are no restrictions on the method for generating segment image data. For example, it may be created by dividing distance images or point cloud data measured using LiDAR (Light detection and ranging) or a stereo camera into segments of a certain distance. Alternatively, it may be created using a direct TOF (Time of Flight) method with an image sensor. The direct TOF method measures the distance to an object by utilizing the time it takes for light emitted from a light source to reach each pixel. Segment images created by the direct TOF method are, for example, images taken when exposed for the elapsed time corresponding to each segment after the light source has emitted light.

[0107] <Other Embodiments Beyond Image Processing> The input data targeted by this method is not limited to images. The technology disclosed herein is confined to high-speed data processing methods. Due to the characteristics of the technology disclosed herein, the input data can be in the form of a graph with adjacency relationships between data points.

[0108] In graph-structured data where multiple data points have adjacency relationships, it is necessary to analyze connected components formed by adjacent nodes with the same value at high speed. In view of this, this disclosure enables high-speed separation detection of connected components when they overlap in graph-structured data.

[0109] In this disclosure, the term "shrinkage," which is common in image processing, is defined in an extended sense to apply to other data formats as follows: In graph-structured data where multiple data points are adjacent to each other, "shrinkage" refers to the process of updating the value of a data point of interest with the smallest value among all the values ​​of the data points adjacent to that data point of interest. Examples of graph-structured data include network data, tactile sensing data, thermography, 3D point cloud data, and voxel data.

[0110] The mark detection system 100 described in the above-described embodiment is an example of a connected component detection system in this disclosure. For example, in the configuration of Figure 5, graph structure data is input to the input unit 10 instead of image data, and the position information of connected components is output from the output unit 40. In the above-described embodiment, the two-dimensional image data corresponds to the graph structure data, and the pixel of interest and the eight neighboring pixels adjacent to it above, below, left, right, and diagonally have an adjacency relationship.

[0111] (Application to tactile sensing data) As an example other than images, this method can also be applied to pressure distribution data measured by tactile sensors used in the fingertips of robot hands, etc. For example, as shown in Figure 28, such pressure distribution data is data arranged in a two-dimensional array, and each data contains the value of the applied pressure. After binarizing this data at a predetermined threshold, similar to image data, the contraction process is performed iteratively, similar to this method. This makes it possible to separate and detect the positions of multiple pressurized points, even if they overlap on the pressure value map obtained from tactile sensor data.

[0112] (Application to thermographic data) In thermographic data, objects such as people may overlap. By using this method, overlapping people and other objects can be separated and detected.

[0113] (Application to Voxel Data) This method can also be applied to 3D data. For example, with voxel data, a 3x3x3 area centered on the voxel of interest is used as the adjacent area. If there is a value of 0 in this area, indicating that there is no background or object, the voxel of interest is updated to 0.

[0114] (Application to Network Data) As another example, this method can also be applied to network data. Although such data is not arranged in a two-dimensional array like an image, for example, as shown in Figure 29, the process of "updating a node of interest with a value of 1 to a value of 0 if any of its neighboring nodes have a value of 0" is called the "collapse process." In this case, as shown in Figure 30, by repeating the collapse process, the network graph collapses, and the locations of important nodes can be identified. This method has potential applications, for example, in community detection in data such as SNS connection relationships and web page link structures.

[0115] (Application to Chemical Composition Data of Industrial Product Parts and Substances) The chemical composition of industrial product parts and substances can also be stored as graph data and this method can be applied to it. This can be used, for example, to search for similar products / substances, and can be valuable in streamlining the development and manufacturing of products / substances.

[0116] (Application to process data in factories) It is also possible to apply this method to graph-like data created by linking the various processes carried out in the manufacturing of a product with the processes carried out before and after it. For example, this can be valuable in making process management easier and manufacturing more efficient by searching for product varieties with similar process flows.

[0117] This disclosure is useful, for example, in speeding up object positioning systems, because it can quickly separate and detect overlapping marks in an image.

[0118] 10 Input unit 11 Image input unit 12 Shrinkage parameter input unit 20 Signal processing unit 30 Processing result determination unit 40 Output unit 100 Mark detection system 300 Positioning system 310 Imaging unit 320 Control unit 330 Movement unit

Claims

1. A mark detection system for detecting the positions of multiple overlapping geometric marks contained in an input image, comprising: an input unit having an image input unit for inputting the input image and a shrinkage parameter input unit for inputting parameters related to shrinkage processing; a signal processing unit that repeatedly performs the shrinkage processing on the input image using the parameters and separates and detects the multiple geometric marks based on the results of this execution; and an output unit that outputs the processing results of the signal processing unit.

2. A mark detection system according to claim 1, wherein the signal processing unit extracts the position coordinates of vanishing points that have disappeared during the process of repeating the contraction process, classifies the vanishing points from which the position coordinates have been extracted based on the number of contraction processes until they disappear, and determines the position coordinates of geometric marks from the position coordinates of the vanishing points in each class.

3. A mark detection system according to claim 2, wherein the parameter includes a threshold value relating to the number of contraction processes until disappearance for classifying vanishing points.

4. A mark detection system according to claim 1, wherein the parameter includes the number of geometric marks included in the input image.

5. A mark detection system according to claim 1, wherein the parameter includes the maximum number of repetitions of the shrinkage process.

6. A mark detection system according to claim 1, wherein the input unit and the output unit are equipped with a display having a user interface screen, and the user interface screen includes an area for displaying the input image, an area for setting the parameters, and an area for displaying the results of the separation detection process of the plurality of geometric marks.

7. A mark detection system according to claim 1, comprising a processing result determination unit that determines the processing result of the signal processing unit, wherein when the processing result determination unit determines that reprocessing is necessary, the parameters are changed and the signal processing unit separates and detects the plurality of geometric marks again.

8. A mark detection system according to claim 7, wherein the maximum number of reprocessing operations for separation detection is configured to be configurable.

9. A positioning system for positioning a first object marked with a first geometric mark and a second object marked with a second geometric mark, comprising: a mark detection system for separating and detecting the first and second geometric marks from an image of the first and second geometric marks; a moving unit for moving the position of at least one of the first and second objects; and a control unit for controlling the amount of movement by the moving unit based on the output of the mark detection system, wherein the mark detection system comprises: an input unit having an image input unit for inputting the image and a shrinkage parameter input unit for inputting parameters related to shrinkage processing; a signal processing unit for repeatedly performing the shrinkage processing on the image using the parameters and separating and detecting the first and second geometric marks based on the results of this execution; and an output unit for outputting the processing results of the signal processing unit.

10. The positioning system according to claim 9, wherein the mark detection system determines whether the first and second geometric shape marks overlap in the image, and if it determines that they do not overlap, the positioning system detects the position coordinates of the first and second geometric shape marks without using the shrinkage process.

11. A mark detection method for detecting the positions of multiple overlapping geometric marks contained in an input image, comprising: a step of repeatedly performing a shrinking process on the input image; a step of extracting the position coordinates of vanishing points that have disappeared during the process of repeating the shrinking process; a step of classifying the vanishing points from which the position coordinates have been extracted based on the number of shrinking processes performed until they disappear; and a step of determining the position coordinates of geometric marks from the position coordinates of the vanishing points in each class.

12. A mark detection method according to claim 11, wherein the number of the plurality of geometric marks is set as the number of classes for classifying the vanishing points.

13. A mark detection method according to claim 11, wherein, as a preprocessing step, at least one of the number of geometric marks, a threshold for classifying vanishing points, and the maximum number of repetitions of the shrinkage process is input.

14. A program for causing a computer to perform the mark detection method described in claim 11.

15. A connected component detection system for detecting the positions of multiple connected components that overlap with each other, contained in input data, comprising: an input unit having a data input unit for inputting the input data and a shrinkage parameter input unit for inputting parameters related to shrinkage processing; a signal processing unit that repeatedly performs the shrinkage processing on the input data using the parameters and separates and detects the multiple connected components based on the results of this execution; and an output unit that outputs the processing results of the signal processing unit.

Images