Contour extraction system and contour extraction method

Abstract

A contour extraction system (10) for extracting a contour of an object (100) from a captured image of the object (100) is characterized by comprising: an imaging device (14) that captures an image of the object (100); an irradiation unit (16) that irradiates the object (100), the irradiation unit (16) being capable of switching irradiation light to a first light having a first color or to a second light having a second color different from the first light; and a controller (20) that calculates the contour of the object (100) from the captured image, wherein the controller (20) is configured to extract the contour of the object (100) in the image on the basis of a comparison result between a luminance value Lr in a first image (30R) obtained by imaging under the first light and a luminance value Lb in a second image (30B) obtained by imaging under the second light.

Description

Contour extraction system and contour extraction method

[0001] This specification discloses a contour extraction system and a contour extraction method for extracting the contour of an object in an image obtained by imaging the object.

[0002] Conventionally, many techniques for extracting the contour of an object based on an image obtained by imaging the object have been proposed. For example, a technique for extracting the contour of an object by binarizing an image obtained by imaging the object with a luminance value is known.

[0003] However, when only binarized with a luminance value, there were cases where the contour of the object could not be clearly extracted due to the influence of the shadow of other members, dirt, etc. Therefore, in some cases, a technique for improving the extraction accuracy of the contour by changing the color of the light irradiated on the object has been proposed.

[0004] For example, Patent Document 1 discloses a technique for changing the color of light irradiated on a visual field according to inspection content. Further, Patent Document 2 discloses a technique for imaging an object a plurality of times by changing the color of light, binarizing each image, and synthesizing the obtained plurality of binarized images. Furthermore, Patent Document 3 discloses a technique for imaging an object by changing the color of light, detecting an inspection region based on one of the obtained two images, and detecting the presence or absence of a defect based on the other.

[0005] Japanese Patent No. 4234661 Japanese Patent No. 2720759 Japanese Patent No. 6759699

[0006] However, it has been difficult to accurately detect the contour of an object with the techniques of Patent Documents 1 to 3. Therefore, this specification discloses a contour extraction technique capable of more accurately detecting the contour of an object.

[0007] The contour extraction system disclosed herein is a contour extraction system for extracting the contour of an object from an image of the object, comprising: an imaging device for imaging the object; an illumination unit for illuminating the object, the illumination unit being able to switch the illumination light between a first light having a first color and a second light of a second color different from the first light; and a controller for calculating the contour of the object from the image, wherein the controller is configured to extract the contour of the object in the image based on a comparison result between the brightness value in a first image obtained by imaging under the first light and the brightness value in a second image obtained by imaging under the second light.

[0008] In this case, the controller may be configured to store in advance as a reference value the correspondence between the brightness value of an image obtained by imaging an object of the same quality as the object under the first light and the brightness value of an image obtained by imaging the same object under the second light, calculate as a corresponding value the correspondence between the brightness value of each of the multiple pixels in the first image and the brightness value of each of the multiple pixels in the second image, and extract the contour of the object in the image based on a comparison between the reference value and the corresponding value.

[0009] Furthermore, the controller may be configured to calculate the similarity between the reference value and the corresponding value for each of the multiple pixels, and to extract the contour of the object in the image based on the binarized image obtained by binarizing the image based on the similarity.

[0010] Furthermore, at least one of the first color and the second color may be a color of the same type as the color of the object, but may not be a color of the same type as the color of any non-objects that appear in the image together with the object.

[0011] In this case, the object is made of gold or copper, the non-object is made of aluminum, the first color is a reddish color, and the second color may be a blued color.

[0012] Furthermore, the object may be a crimp ball formed on a semiconductor device, and the non-object may be an aluminum pad to which the crimp ball is pressed.

[0013] Furthermore, the object is a crimp ball formed on a semiconductor device, and the non-object that appears in the image together with the object is an aluminum pad to which the crimp ball is crimped. The controller may be configured to extract the contour of the ball neck interposed between the crimp ball and the bonding wire based on the first image or the second image, and to consider the corresponding value inside the contour of the ball neck as the same as the reference value.

[0014] Furthermore, the contour extraction method for extracting the contour of an object from an image of the object is characterized by: acquiring a first image by illuminating the object with a first light having a first color; acquiring a second image by illuminating the object with a second light of a second color different from the first color; and extracting the contour of the object in the image based on the comparison result between the brightness value in the first image and the brightness value in the second image.

[0015] The technique disclosed herein allows for more accurate detection of the contour of an object by utilizing the comparison result between the brightness value of the first image and the brightness value of the second image.

[0016] This is a schematic diagram showing the configuration of the contour extraction system. This is a diagram showing an example of the first and second images. This is a diagram showing the first image with explanatory lines added. This is a diagram showing the correspondence between the luminance values ​​in red light and the luminance values ​​in blue light for gold and aluminum, respectively. This is a diagram showing a reference value and an example of similarity C based on the reference value. This is a schematic diagram showing the contour extraction process. This is a diagram showing an example of a normalized image. This is a diagram showing an example of a binarized image. This is a flowchart showing the contour extraction process.

[0017] The configuration of the contour extraction system 10 will be described below with reference to the drawings. Figure 1 is a schematic diagram showing the configuration of the contour extraction system 10. This contour extraction system 10 images an object 100 and extracts the contour of the object 100 based on the obtained image. In this example, the contour extraction system 10 extracts the contour of a crimp ball 102 formed on a semiconductor device as the object 100. As will be explained in detail later, the crimp ball 102 is a part in which the end of a bonding wire connecting electrodes is crimped to a pad 112 that functions as an electrode.

[0018] The contour extraction system 10 includes a microscope 12, an imaging device 14, an illumination unit 16, and a controller 20. The microscope 12 magnifies the image of the fine crimped ball 102 so that it can be observed with the naked eye. Such a microscope 12 is, for example, an optical microscope. The imaging device 14 captures the image of the crimped ball 102 obtained by the microscope 12 and outputs image data. Such an imaging device 14 has an image sensor such as a CMOS or CCD. In this example, the imaging device 14 is a monochrome imaging device that generates grayscale image data. However, naturally, the imaging device 14 may also be a color imaging device that generates color image data. The image captured by this imaging device 14 is transmitted to the controller 20.

[0019] The irradiation unit 16 irradiates light onto the object 100, which is the crimped ball 102. The irradiation unit 16 may be incorporated inside the microscope 12 or placed outside the microscope 12. In either case, the irradiation unit 16 is positioned and oriented so as to irradiate the entire object 100.

[0020] The irradiation unit 16 is capable of emitting at least a first light, which is a first color, and a second light, which is a second color different from the first color. The colors of the first and second lights are not particularly limited, as long as they are different from each other. Typically, the first and second colors are colors that are clearly recognized as "different colors." For example, the first and second colors are colors that are located in different regions when the color wheel is divided into three or four equal parts. In this example, a reddish color is used as the first color, and a blued color is used as the second color. Therefore, the irradiation unit 16 in this example is capable of emitting red and blue light. In order to selectively emit two different colors of light, the irradiation unit 16 may have two types of light sources. For example, the irradiation unit 16 may have a red LED and a blue LED. Alternatively, the irradiation unit 16 may have a color filter placed in the optical path of the light from the light source that changes the color of the transmitted light. In any case, the irradiation unit 16 changes the color of the light irradiated onto the crimping ball 102 in response to a command from the controller 20.

[0021] The controller 20 is physically a computer having a processor 22 and memory 24. Although Figure 1 shows the controller 20 as a single computer, the controller 20 may be configured by combining multiple physically separated computers. In addition to the processor 22 and memory 24, the controller 20 may also have a communication interface for exchanging data with external devices and a user interface for presenting information to the user or receiving operation instructions from the user.

[0022] The controller 20 analyzes the image captured by the imaging device 14 and extracts the contour of the object 100, which is the crimped ball 102. In order to accurately extract the contour, the controller 20 drives the imaging device 14 and the illumination unit 16 to acquire a first image 30R captured under red light illumination and a second image 30B captured under blue light illumination. The controller 20 then extracts the contour of the object 100 by comparing the brightness values ​​Lr and Lb of the first image 30R and the second image 30B, respectively. The principle of this contour extraction will be explained below.

[0023] First, let's describe the crimp ball 102, which is the object 100 in this example. The crimp ball 102 is formed by melting the end of the bonding wire 106 to form a sphere, which is then pressed onto a metal plate called a pad 112. Therefore, the crimp ball 102 itself is a flat disc shape. There is also a roughly conical portion between the crimp ball 102 and the bonding wire 106 that is formed when pressed with a capillary (not shown). Hereafter, this roughly conical portion will be referred to as the ball neck 104. In this example, the pad 112 is made of aluminum, and the bonding wire 106, which includes the crimp ball 102, is made of gold or copper. Therefore, the pad 112 is white or silver, and the crimp ball 102 is reddish-yellow or reddish-brown. Below, we will explain using the case where the crimp ball 102 is made of gold as an example.

[0024] Figure 2 shows an image obtained by imaging the crimp ball 102 with the imaging device 14. The left image in Figure 2 is the first image 30R, which was captured under red light illumination, and the right image is the second image 30B, which was captured under blue light illumination. Figure 3 is the first image 30R with explanatory lines added. As described above, in this example, the imaging device 14 captures a grayscale image. The contour extraction system 10 extracts the contour of the crimp ball 102 from these images 30R and 30B and detects the shape and diameter of the crimp ball 102. The detected shape and diameter are important indicators for evaluating the quality of the semiconductor device.

[0025] Here, images 30R and 30B are grayscale images and are composed of multiple pixels with brightness values ​​corresponding to the reflected light from the subject. Image 30 shows the pad 112 and the crimping ball 102. Inside the crimping ball 102, there is a roughly circular image (see Figure 3) corresponding to the ball neck 104. Furthermore, the shadow of the bonding wire 106 is visible diagonally to the upper left from the ball neck 104. If there is such a shadow of the bonding wire 106 or dirt on the pad 112, the boundary between the crimping ball 102 and the pad 112 becomes unclear, and the outline of the crimping ball 102 cannot be properly extracted. This problem occurs not only in the first image 30R but also in the second image 30B.

[0026] Therefore, in order to accurately extract the contour of the object 100, the contour extraction system 10 in this example obtains the correspondence (e.g., ratio, etc.) between the luminance value Lr in the first image 30R and the luminance value Lb in the second image 30B as the correspondence value RB, and compares it with a pre-stored reference value RBref. The reference value RBref is a value that shows the correspondence between the luminance value Lr in red light and the luminance value Lb in blue light of an object of the same quality as the object 100.

[0027] The correspondence between the two luminance values ​​Lr and Lb varies greatly depending on the properties of the material, particularly its color. This will be explained with reference to Figure 4. In Figure 4, the solid lines show the luminance values ​​Lr and Lb when an object made of gold is imaged, and the dashed lines show the luminance values ​​Lr and Lb when aluminum is imaged.

[0028] Even when imaging the same object, the brightness value L of the resulting image will differ depending on the color of the light used for illumination. For example, suppose that when a gold object is imaged under red light, the brightness value of one pixel P1 is Lr1 and the brightness value of another pixel P2 is Lr2. Subsequently, when this gold object is imaged under blue light, the brightness value of one pixel P1 becomes Lbg1, which is significantly lower than Lr1, and the brightness value of another pixel P becomes Lbg2, which is significantly lower than Lr2. This is because the surface color of a gold object contains a red component, and while it reflects a lot of long-wavelength light in the visible range (light that appears red), it easily absorbs short-wavelength light (light that appears blue).

[0029] On the other hand, silvery objects like aluminum reflect a lot of both long-wavelength and short-wavelength light. Therefore, the brightness value L obtained from an aluminum object does not change significantly whether it is illuminated with red or blue light. For example, suppose that when an aluminum object is imaged under red light, the brightness value of one pixel P1 is Lr1 and the brightness value of another pixel P2 is Lr2. Subsequently, when this aluminum object is imaged under blue light, the brightness value of one pixel P1 becomes Lba1, which is approximately the same as Lr1, and the brightness value of another pixel P becomes Lba2, which is approximately the same as Lr2.

[0030] Thus, the correspondence between luminance values ​​for two types of light (i.e., the correspondence value RB) differs depending on the material. Therefore, in this example, the correspondence between an object of the same quality as the object 100 is stored in memory 24 as the reference value RBref. In this example, since the object 100 is a gold crimped ball 102, the reference value RBref is the correspondence between luminance values ​​Lr and Lb obtained by imaging a material of the same quality as the object 100, i.e., a gold object, under red and blue light. Such a reference value RBref may also be obtained, for example, by plotting points corresponding to the luminance values ​​Lr and Lb for each pixel P of an image obtained by imaging a gold object, generating a scatter plot, and obtaining the approximate curve of this scatter plot as the reference value RBref.

[0031] The controller 20 sets the similarity score C based on the reference value RBref. Figure 5 shows the reference value RBref and an example of the similarity score C based on the reference value RBref. The controller 20 also calculates the correspondence between the luminance value Lr in red light and the luminance value Lb in blue light (i.e., the corresponding value RB) for the object 100, and calculates the similarity score C with respect to the reference value RBref. The similarity score C is an index that represents the strength of the relationship between the reference value RBref and the corresponding value RB. The formula for calculating the similarity score C is not particularly limited, as it changes according to the degree of similarity of the corresponding value RB with respect to the reference value RBref. For example, the similarity score C may change linearly or non-linearly, taking "1" when the corresponding value RB matches the reference value RBref, and approaching 0 as it moves away from the reference value RBref.

[0032] For example, in the case of the example in Figure 5, if the corresponding value RB of a particular pixel P1 in the image 30 of the object 100 is point M1 in Figure 5, then the similarity C of that pixel P is 1. Also, if the corresponding value RB of another pixel P2 is M2 in Figure 5, then the similarity C of that other pixel P is 0.5, and if it is M3 in Figure 5, then the correlation coefficient of that other pixel P is 0.

[0033] The controller 20 calculates the similarity C for all pixels P that make up the image 30 of the object 100. Then, based on the obtained similarity C, the controller 20 binarizes the image 30 and extracts the contour of the object 100 by extracting edges from the resulting binarized image.

[0034] For example, suppose that a first image 30R and a second image 30B are obtained for the object 100, as shown in the upper part of Figure 6. In Figure 6, for the sake of simplicity, image 30 is composed of 9 pixels in a 3x3 grid. Also, in the first image 30R, the brightness value L of all 9 pixels is Lr1. On the other hand, in the second image 30B, the brightness value L varies in the range of Lb1 to Lb4.

[0035] In this case, as shown in the middle right of Figures 5 and 6, the similarity C of pixel P in Lb1 is "1", the similarity C of pixel P in Lb2 is "0.8", the similarity C of pixel P in Lb3 is "0.2", and the similarity C of pixel P in Lb4 is "0". Hereafter, the data to which the similarity C has been applied to each pixel P will be referred to as the "similarity image 31".

[0036] The controller 20 further normalizes the similarity image 31 to a range of 0 to 255. Hereafter, this normalized data will be referred to as the "normalized image 40". The middle left of Figure 6 shows the normalized image 40. The controller 20 further binarizes the normalized image 40. The bottom of Figure 6 shows the binarized image of the normalized image 40. The controller 20 extracts the contour of the object 100 based on the obtained binarized image 42. In the example in Figure 6, line K is extracted as the contour of the object 100.

[0037] Figures 7 and 8 show the normalized image 40 and the binarized image 42 calculated from the first image 30R and the second image 30B of Figure 2. As is clear from the comparison of Figures 2 and 7, in the normalized image 40 derived from the similarity C between the corresponding value RB and the reference value RBref, the effects of the shadow of the wire 106 and the dirt on the pad 112 are eliminated, and the boundary between the crimp ball 102 and the pad 112 is clearly defined. In the example of Figure 7, the inner portion of the ball neck 104 is pre-processed with a similarity C=1. That is, the portion corresponding to the ball neck 104 appears darker than the crimp ball 102, and the boundary of the ball neck 104 can be clearly distinguished to some extent in the first image 30R and the second image 30B. Therefore, prior to generating the similarity image 31, the controller 20 extracts the contour of the ball neck 104 based on the first image 30R or the second image 30B. The controller 20 then generates a similarity image 31, and subsequently a normalized image 40, by setting the similarity C = 1 for the pixels P inside the contour.

[0038] The controller 20 binarizes the normalized image 40 shown in Figure 7 and extracts the contour of the crimped ball 102. Figure 8 is the binarized image 42 generated from the normalized image 40. The controller 20 extracts the contour of the crimped ball 102 based on this binarized image 42. The controller 20 may perform general image processing, such as histogram averaging, blob processing, and morphological transformation processing, before or after the binarization process. Furthermore, the controller 20 may perform general preprocessing, such as sharpening, tone adjustment, and filtering, on the first image 30R and the second image 30B before generating the similarity image 31 and the normalized image 40.

[0039] Next, the flow of the contour extraction process of the crimped ball 102 by the contour extraction system 10 will be explained with reference to Figure 9. In this case, the contour extraction system 10 has in advance stored a reference value RBref and a similarity value C to the reference value RBref in the memory 24, as shown in Figure 5. When the contour extraction system 10 extracts the contour of the crimped ball 102, it first acquires a first image 30R (S10). That is, the controller 20 irradiates the illumination unit 16 with red light and causes the imaging device 14 to image the crimped ball 102. Subsequently, the controller 20 irradiates the illumination unit 16 with blue light and causes the imaging device 14 to image the crimped ball 102, acquiring a second image 30B (S12).

[0040] The controller 20 binarizes the first image 30R or the second image 30B and extracts the contour of the ball neck 104 based on the obtained binarized image (S14). Subsequently, the controller 20 generates a similarity image 31 and a normalized image 40 based on the first image 30R and the second image 30B (S16, S18). That is, for each pixel P, the controller 20 applies the brightness value Lr of the first image 30R and the brightness value Lb of the second image 30B to the graph in Figure 5 and calculates the similarity C. However, for the area inside the contour of the ball neck 104, S = 1.0 regardless of the brightness values ​​Lr and Lb. Subsequently, the controller 20 normalizes the similarity C calculated for each pixel P between 0 and 255 and calculates the normalized image 40. Subsequently, the controller 20 binarizes the normalized image 40 using the same procedure as for normal contour extraction (S20), and then extracts areas where the brightness value L changes from the binarized image 42 as contours of the object 100 (S22).

[0041] As is clear from the above explanation, in this example, a first image 30R captured under red light and a second image 30B captured under blue light are acquired, and a binarized image 42 is generated according to the similarity C between the corresponding value RB of the two images 30R and 30B and the reference value RBref, thereby enabling accurate extraction of the object 100 in the image.

[0042] It should be noted that the configurations described above are all examples, and other configurations may be changed as appropriate, as long as the configuration described in claim 1 is met. For example, in the above description, red and blue light are used for illumination, but the color of the light used for illumination may be changed as appropriate. For example, the color of the light used for illumination may be changed according to the colors of the object 100 and the non-object 110 that are captured in the image together with the object 100. For example, when illuminating with a first color and a second color, the first color may be a color of the same type as the object 100, but not a color of the same type as the surface color of the non-object 110. Furthermore, in this case, the second color may be a color that is not of the same type as the surface color of either the object 100 or the non-object 110. Alternatively, the second color may be a color of the same type as the non-object 110, but not a color of the same type as the surface color of the object 100. "Similar colors" are colors that are located in the same area when the color wheel is divided into three or four equal parts, while "non-similar colors" are colors that are located in different areas when the color wheel is divided into three or four equal parts.

[0043] Furthermore, although the above explanation assumes two types of light colors are used, three or more colors of light may be used. For example, a first image, a second image, and a third image may be obtained by irradiating with light of three different colors, and these three images may be combined. The contour obtained from comparing the brightness values ​​of the first and second images may be combined with the contour obtained from comparing the brightness values ​​of the first and third images.

[0044] Also, in the above description, the bonding ball 102 is treated as the object 100. However, the technology disclosed in this specification may be applied to the contour extraction of objects other than the bonding ball 102. Also, in this example, the object 100 is imaged in grayscale, but it may be imaged in color. Further, the above-described procedure is an example. If the comparison result between the luminance value Lr of the first image 30R and the luminance value Lb of the second image 30B is used, the contour extraction procedure may be appropriately changed. For example, for each pixel P, the ratio of the luminance value Lr to the luminance value Lb may be calculated as the luminance ratio, and the binarized image 42 may be generated based on the luminance ratio. That is, since the yellow luminance ratio Lb / Lr of gold is around 0.3 and the luminance ratio Lb / Lr of aluminum is around 1, even if the luminance ratio is binarized with a value around 0.6, the gold region can be distinguished from aluminum, and the contour of the gold object can be extracted.

[0045] Also, in the above description, the luminance values Lr and Lb compare the pixel P of the first image 30R and the second image 30B, respectively. However, it may be a region combining a plurality of pixels P of the first image 30R and the second image 30B, and for example, the statistical values (such as the average value or the median value) of the luminance values of the plurality of pixels included in each combined region may be compared.

[0046] 10 Contour extraction system, 12 Microscope, 14 Imaging device, 16 Irradiation unit, 20 Controller, 22 Processor, 24 Memory, 30B Second image, 30R First image, 31 Similarity image, 40 Normalized image, 42 Binarized image, 100 Object, 102 Bonding ball, 104 Ball neck, 106 Bonding wire, 110 Non-object, 112 Pad.

Claims

1. A contour extraction system for extracting the contour of an object from an image of the object, comprising: an imaging device for imaging the object; an illumination unit for illuminating the object, the illumination unit being able to switch the illumination light between a first light having a first color and a second light of a second color different from the first light; and a controller for calculating the contour of the object from the image, wherein the controller is configured to extract the contour of the object in the image based on a comparison result between the brightness value in a first image obtained by imaging under the first light and the brightness value in a second image obtained by imaging under the second light.

2. A contour extraction system according to claim 1, wherein the controller pre-stores as a reference value the correspondence between the brightness value of a reference image obtained by imaging an object identical to the object under the first light and the brightness value in the reference image under the second light, calculates as a corresponding value the correspondence between the brightness value of each of the multiple regions of the first image and the brightness value of each of the multiple regions of the second image, and extracts the contour of the object in the image based on a comparison between the reference value and the corresponding value.

3. A contour extraction system according to claim 2, wherein the controller is configured to calculate the similarity between the reference value and the corresponding value for each of a plurality of pixels, and to extract the contour of the object in the image based on the binarized image obtained by binarizing the similarity.

4. A contour extraction system according to claim 1, characterized in that at least one of the first color and the second color is a color of the same type as the color of the object, while being a color of a non-object that is captured in the image together with the object.

5. A contour extraction system according to claim 4, wherein the object is an object made of gold or copper, the non-object is an object made of aluminum, the first color is a reddish color, and the second color is a blued color.

6. A contour extraction system according to claim 5, wherein the target object is a crimp ball formed on a semiconductor device, and the non-target object is an aluminum pad to which the crimp ball is crimped.

7. A contour extraction system according to claim 3, wherein the object is a crimp ball formed on a semiconductor device, the non-object that appears in the image together with the object is an aluminum pad to which the crimp ball is crimped, and the controller is configured to extract the contour of a ball neck interposed between the crimp ball and a bonding wire based on the first image or the second image, and to consider the corresponding value inside the contour of the ball neck as the same as the reference value.

8. A contour extraction method for extracting the contour of an object from an image of the object, characterized by: acquiring a first image by photographing the object while irradiating it with a first light having a first color; acquiring a second image by photographing the object while irradiating it with a second light of a second color different from the first color; and extracting the contour of the object in the image based on a comparison result between the brightness value in the first image and the brightness value in the second image.

Images