Apparatus for acquiring three-dimensional information of objects and surfaces for an artificial vision system for the automatic optical inspection of the visual quality of an underlying object

Abstract

The apparatus (1) comprises an upper camera (10) and a plurality of side cameras (20) arranged around the vertical optical axis (Z) of the upper camera (10); a top-lighting system for the object (100) comprising a plurality of light projectors (30a, 30b, 30c, 30d) of different colors or wavelengths; and an electronic controller adapted to command the simultaneous emission of light beams (32a, 32b, 32c, 32d) of different colors or wavelengths from the light projectors (30a, 30b, 30c, 30d), each camera (10, 20) is provided with spatial separation means (120) for the reflected light beams (32'a, 32'b, 32'c, 32'd) of different colors or wavelengths reflected from the object (100), and with a plurality of optical sensors (110a, 110b, 110c, 110d) configured to simultaneously acquire the reflected light beams (32'a, 32'b, 32'c, 32'd) of different colors or wavelengths separated by the separation means (120).

Description

[0001] APPARATUS FOR ACQUIRING THREE-DIMENSIONAL INFORMATION OF OBJECTS AND SURFACES FOR AN ARTIFICIAL VISION SYSTEM FOR THE AUTOMATIC OPTICAL INSPECTION OF THE VISUAL QUALITY OF AN UNDERLYING OBJECT DESCRIPTION

[0002] The present invention relates to an apparatus for acquiring three-dimensional information of objects and surfaces for an artificial vision system intended for the automatic optical inspection of objects, in particular, but not limited to, electronic assemblies, electronic boards, and similar items. It is known that artificial vision systems for visual quality inspection are widely applied in high-volume manufacturing industries, in the semiconductor industry, and in the food and pharmaceutical industries and they are based on standard image processing and computer vision techniques such as, but not limited to, edge detection, connected component analysis, texture analysis, and projective geometry.

[0003] These approaches are simple and very effective when it is necessary to perform quantitative measurements of well-defined entities (such as lengths, widths, colors, or fine-grain patterns); once measurements are completed, simple rule-based tools can be used to evaluate whether the observed product meets the acceptance criteria.

[0004] In this field, the expression “Automatic Optical Inspection” (AOI) generally refers to an automated visual inspection system for assessing the quality of objects (which may consist of electronic assemblies, such as printed circuit boards, PCBs and Surface Mount Technology, SMT), in which a camera autonomously scans the object under test.

[0005] In particular, in the case of electronic assemblies, the camera allows the identification of both manufacturing defects (e.g., missing components) and quality defects (e.g., size or joint shape or component misalignment). AOI systems are commonly used in manufacturing processes because they provide non-contact testing and inspection methods; AOI systems are implemented at manystages of the production process, including bare board inspection, solder paste inspection (SPI), pre-reflow and post-reflow inspection, among others.

[0006] As is well known, virtually all automatic optical inspection systems require projecting light, or one or more structured light patterns, onto the object to be inspected and acquiring the light reflected from the object through a digital sensor; the acquired images are analysed by a processing unit configured to determine the physical and / or geometric characteristics of the object to be inspected based on the light captured by the sensor.

[0007] Nowadays, in the field of automated visual quality inspection systems for electronic assemblies, there is a growing need to acquire coordinated in-line measurements.

[0008] As the complexity of modem electronic boards continues to increase (due to the use of more components, more joints, higher component density, and new packaging technologies), traditional two-dimensional automatic optic inspection technology which relies on grayscale or color image analysis from side cameras may no longer be a viable option.

[0009] To overcome these limitations, three-dimensional scanning technology has been effectively combined with AOI and is now used in many applications, such as the inspection of microelectronic components and solder paste deposits below 100 microns, as well as other demanding use cases.

[0010] However, although functional, these known systems have certain drawbacks and limitations. In particular, some limitations arise from the nature of the measurement technique itself, while others are more specifically related to the inspection of electronic assemblies (SMT and PCB), and include:

[0011] - difficulty in ensuring complete measurement of low components located near tall ones due to shadowing effects (when the reference pattern is projected at an angle, taller features can cast shadows that prevent measurement of adjacent lower features);- difficulty in avoiding measurement errors caused by multiple reflections between components (multiple specular reflections among shiny elements such as solder joints, tinned wires, and metallic oscillators can cause distortions in the fringe pattern and errors in height measurements); - difficulty in guaranteeing fast, highly accurate, and repeatable measurements in the micrometer (pm) range in all directions.

[0012] A further difficulty lies in the physical complexity and bulkiness of the imaging acquisition systems used in existing solutions, which require relatively large dimensions for the complete inspection system and to ensure that different components do not interfere with each other.

[0013] There is therefore a need to improve the structure of artificial vision systems for visual quality inspection known in the art.

[0014] The technical objective of the present invention is therefore to provide an apparatus for acquiring three-dimensional information of objects and surfaces for an artificial vision system intended for the automatic optical inspection of the visual quality of manufactured products, which overcomes the above-mentioned drawbacks of the prior art.

[0015] Within the scope of this technical task, one objective of the invention is to provide an apparatus for acquiring three-dimensional information of objects and surfaces for an artificial vision system for automatic optical inspection of visual quality, which is extremely accurate, fast, and repeatable. Another objective of the invention is to provide an apparatus for acquiring three-dimensional information of objects and surfaces for an artificial vision system for automatic optical inspection of visual quality, capable of maintaining high accuracy and speed even when measuring objects having a wide range of height dimensions.

[0016] The technical task, as well as these and other objectives, are achieved according to the present invention by providing an apparatus for acquiring three-dimensional information of objects and surfaces for an artificial vision system for the automatic optical inspection of the visual quality of an underlying object, characterized in that it comprises an upper camera having a vertical opticalaxis, a plurality of side cameras arranged around the vertical optical axis of said upper camera, a lighting system from above the object comprising a plurality of light projectors of different colors or wavelengths arranged around the vertical optical axis of said upper camera, an electronic controller adapted to command the simultaneous emission of light beams of different colors or wavelengths from said plurality of light projectors, each camera comprising spatial separation means for the light beams of different colors or wavelengths reflected from the object, and a plurality of optical sensors for the simultaneous acquisition of the reflected and separated light beams of different colors or wavelengths by the separation means.

[0017] It has been found particularly advantageous to use projectors configured to project four different colors or wavelengths in combination with high-resolution cameras each integrating four optical sensors for acquiring the four different colors or wavelengths.

[0018] It has also been found particularly advantageous to project simultaneously four different colors or wavelengths decomposed onto the four optical sensors of the cameras, where the cameras are high-resolution and the colors are Red, Green, Blue, and NIR (R, G, B, NIR).

[0019] The innovative approach of parallel emission of light beams of different colors allows maintaining 3D vision performance while significantly reducing acquisition times.

[0020] Furthermore, it has been found particularly advantageous to use in each camera four optical sensors of 25 megapixels to inspect a field of view (FoV) of 2500 mm2, thereby increasing the speed of field-of-view acquisition without loss of precision or accuracy.

[0021] The setup of the apparatus according to the invention allows observing each projected beam from different angles, and different combinations of observation angles and projection angles are available for each point in the observed scene.

[0022] By applying image processing algorithms, data analysis algorithms, and data fusion algorithms, it is possible to reconstruct extremely detailed 3D surfaces free from reflections.The apparatus according to the invention, thanks to the combination of a high number of acquired images, advantageously allows reducing or eliminating blind spots and shadowing artifacts, achieving full coverage of the object or surface under observation.

[0023] The apparatus according to the invention can operate under at least three operational conditions: -a first operating lighting condition in which light beams are generated simultaneously by said plurality of projectors, combined with the acquisition of images of the reflected beams by the upper camera;

[0024] -a second operating lighting condition in which light beams are generated simultaneously by said plurality of projectors, combined with the acquisition of images of the reflected beams by said plurality of side cameras;

[0025] -a third operating lighting condition in which light beams are generated by an additional light projector and directed through the optical path of the telecentric lens of the upper camera, combined with the acquisition of images of the reflected beams by the plurality of side cameras. In the first operating condition, four 3D images are obtained; in the second operating condition, sixteen 3D images are obtained; and in the third operating condition, four 3D images and optionally one additional 2D image are obtained if the upper camera is also active.

[0026] With the above combinations, the apparatus according to the invention is thus capable of providing 24 reconstructed 3D images from different viewpoints.

[0027] Other features of the present invention are also defined in the following claims.

[0028] Further characteristics and advantages of the invention will become more apparent from the description of a preferred, though not exclusive, embodiment of the apparatus according to the invention, provided by way of illustrative and non-limiting example in the accompanying drawings, in which:

[0029] Figure 1 shows a schematic elevation view of the apparatus in a first embodiment;

[0030] Figure 2 shows a schematic top plan view of the apparatus in a first embodiment;Figures 3a, 3b, and 3c show the apparatus in three different operating conditions;

[0031] Figure 4 schematically shows one embodiment of the architecture of a side camera for decomposing and acquiring the four colors or wavelengths, though the upper camera may have an entirely similar architecture;

[0032] Figure 5 schematically shows one embodiment of the architecture of the projector for the selective projection of a color or wavelength.

[0033] The following detailed description refers to the accompanying drawings, which form part of this specification.

[0034] In the drawings, similar reference numerals typically identify similar components, unless otherwise indicated by the context.

[0035] The illustrative embodiments described in the detailed description and shown in the drawings are not intended to be limiting.

[0036] Other embodiments may be used, and modifications may be made without departing from the spirit or scope of the subject matter described herein.

[0037] The aspects of the present description, as generally described and illustrated in the figures, may be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are expressly contemplated and form part of this disclosure.

[0038] With reference to the cited figures, there is shown an apparatus for acquiring three-dimensional information of objects and surfaces for an artificial vision system for the automatic optical inspection of the visual quality of an underlying object, generally indicated by reference numeral 1.

[0039] The apparatus 1 comprises cameras 10, 20 for acquiring images of the object 100, in particular an upper camera 10 and a plurality of side cameras 20.The upper camera 10 has a vertical optical axis Z around which the side cameras 20 are arranged. The side cameras 20 are preferably equally angularly spaced around the vertical optical axis Z of the upper camera 10.

[0040] The side cameras 20 preferably have an optical axis that is oblique with respect to the vertical optical axis Z of the upper camera 10.

[0041] Preferably the oblique optical axis of all the side cameras 20 has the same inclination with respect to the vertical optical axis Z of the upper camera 10.

[0042] The apparatus 1 further comprises a top-lighting system for the object 100, comprising a plurality of light projectors 30a, 30b, 30c, 30d of different colors or wavelengths, arranged around the vertical optical axis Z of the upper camera 10.

[0043] The light projectors 30a, 30b, 30c, 30d of different colors or wavelengths are arranged at an intermediate height between the upper camera 10 and the side cameras 20.

[0044] Advantageously the electronic controller of the apparatus 1 is adapted to command the lighting of the object 100 with light beams 32a, 32b, 32c, 32d of different colors or wavelengths emitted simultaneously by the plurality of light projectors 30a, 30b, 30c, 30d.

[0045] Advantageously, moreover, each camera 10, 20 has spatial separation means 120 for the reflected light beams 32’a, 32’b, 32’c, 32’d from the object 100 that were generated by the incident light beams 32a, 32b, 32c, 32d of different colors or wavelengths on the object 100.

[0046] Each camera 10, 20 also advantageously has a plurality of optical sensors 110a, 110b, 110c, 1 lOd, each of which is arranged to acquire, simultaneously with the other optical sensors 110a, 110b, 110c, HOd, a corresponding reflected light beam 32’a, 32’b, 32’c, 32’d that has been separated from the other reflected light beams 32’a, 32’b, 32’c, 32’d.

[0047] The optical sensors 110a, 110b, 110c, HOd are two-dimensional pixel sensors, for example but not necessarily CMOS or CCD sensors, having their respective focal planes aligned.Each optical sensor 110a, 110b, 110c, 1 lOd is independent, as it is associated with a corresponding FPGA unit 37a, 37b, 37c, 37d, so that the images detected by the optical sensors 110a, 110b, 110c, HOd can be acquired in parallel and then stored and processed in an image acquisition and processing unit 38.

[0048] The image acquisition and processing unit 38 of each camera 10, 20 has an output interface for transmitting data to a remote unit, such as a PC or a graphics processing unit (GPU).

[0049] In Figure 4 the separation means 120, in one embodiment, comprise a multichannel optical separator, for example an optical prism 120a, with a corresponding optical sensor 110a, 110b, 110c, HOd associated with each channel. In this case, the optical prism 120a is triangular. Other architectures for the separation means are also possible.

[0050] For example, the separation means may comprise a plurality of optical prisms, for example cubeshaped prisms, each of which is associated with a corresponding sensor 110a, 110b, 110c, 1 lOd. The light projectors 30a, 30b, 30c, 30d of different colors or wavelengths preferably include four light projectors of different colors or wavelengths, for example red, green, blue, and NIR.

[0051] Each light projector 30a, 30b, 30c, 30d, in a preferred embodiment, may be formed by a plurality of LED light sources 30’, 30”, 30’”, 30”” of different colors or wavelengths that can be selectively activated, a lens 31, and a digital micromirror device (Digital Micromirror Display, DMD) 33 interposed between the lens 31 and the LED light sources 30’, 30”, 30”’, 30””.

[0052] Each light projector 30a, 30b, 30c, 30d may have, as shown, four LED light sources 30’, 30”, 30’”, 30”” of different colors or different wavelengths.

[0053] Alternatively, in a simplified version, each light projector 30a, 30b, 30c, 30d may have a smaller number of LED light sources of different colors or wavelengths, for example two LED light sources of different colors or wavelengths. In this case the group of four light projectors of different colors or wavelengths is configured to nonetheless emit simultaneously four light beams of different colors or wavelengths.Between the digital micromirror device 33 and the LED light sources 30’, 30”, 30’”, 30”” there are suitable mirrors 36, each selectively reflecting a respective color or wavelength.

[0054] The digital micromirror device 33 and each LED light source 30’, 30”, 30’”, 30”” can be driven by corresponding drivers DI, D’, D”, D’”, D”” connected to a programmable logic device (FPGA) 35 present on a control electronic board 34.

[0055] In this way it is possible to generate and project any type of light pattern onto the object 100. In a preferred embodiment, the light projectors 30a, 30b, 30c, 30d have a vertical projection axis parallel to the vertical optical axis Z of the upper camera 10, and the lighting system further comprises a plurality of mirror reflectors 3 li interposed between the plurality of projectors 30a, 30b, 30c, 30d and the station for the object 100, for converting the optical paths of the light beams 32a, 32b, 32c, 32d emitted simultaneously by the plurality of projectors 30a, 30b, 30c, 30d. The individual light projectors 30a, 30b, 30c, 30d and the corresponding mirror reflectors 3 li are equally angularly spaced around the vertical optical axis Z of the upper camera 10.

[0056] In a second embodiment, the light projectors 30a, 30b, 30c, 30d have projection axes that are inclined with respect to the vertical axis Z and arranged to converge on the station for the object 100.

[0057] Preferably, the lighting system is configured so that the light beams 32a, 32b, 32c, 32d strike with the same inclination with respect to the vertical direction.

[0058] The apparatus 1 also comprises a telecentric lens 55 having a main vertical optical arm 51 associated with the upper camera 10 and coaxial with the vertical optical axis Z, and a secondary optical arm 52 branching off transversely, in particular horizontally, from the main vertical optical arm 51.

[0059] The optical arms 51, 52 referred to here are in practice optical paths, preferably linear ones, which may be defined by known optical and structural elements.A supplemental light projector 30e is associated with the secondary optical arm 52 of the telecentric lens 55.

[0060] The supplemental projector 30e may have the same architecture as the plurality of projectors 30a, 30b, 30c, 30d.

[0061] The telecentric lens 55 is configured for the acquisition, by the upper camera 10, both of the reflected light beams 32’ a, 32’b, 32’ c, 32’ d generated by the incident light beams 32a, 32b, 32c, 32d of different colors or wavelengths emitted by the light projectors 30a, 30b, 30c, 30d, and of the reflected light beam 32’e generated by the incident light beam 32e emitted by the supplemental light projector 30e.

[0062] To this end, an optical prism 50 is provided at the intersection between the main optical arm 51 and the secondary optical arm 52.

[0063] The optical prism 50 is specifically configured and arranged to direct, into the main optical arm 51 in a downward emission direction the light beam 32e introduced by the supplemental projector 30e into the secondary optical arm 52, and to direct towards the upper camera 10 the reflected light beam 32’e from the object 100 re-entering upward into the main optical arm 51.

[0064] The apparatus 1 may also comprise an indirect lighting system for the object 100. This indirect lighting system preferably comprises a plurality of rings emitting monochromatic or polychromatic light 40i, or a plurality of light emitters arranged along a plurality of rings, the rings being coaxial with the vertical optical axis Z of the upper camera 10 and having increasing diameter and increasing distance away from the upper camera 10.

[0065] The operation of the image acquisition apparatus for inspection of an underlying object according to the invention is evident from what has been described and illustrated and, in particular, is essentially as follows.

[0066] The apparatus 1 provides at least three operating conditions.In a first operating condition, shown in Figure 3 a, lighting with the plurality of projectors 30a, 30b, 30c, 30d is active and image acquisition with the upper camera 10 is active, while neither the supplemental projector 30e nor the side cameras 20 are active.

[0067] In a second operating condition, shown in Figure 3b, lighting with the plurality of projectors 30a, 30b, 30c, 30d is active and image acquisition with the side cameras 20 is active, while neither the supplemental projector 30e nor the upper camera 10 is active.

[0068] In a third operating condition, shown in Figure 3c, lighting with the supplemental light projector 30e is active, which emits a light beam 32e of one color (for example, green) routed through the optical path of the telecentric lens 55, and image acquisition with the side cameras 20 and optionally with the upper camera 10 is active, while lighting with the plurality of projectors 30a, 30b, 30c, 30d is not active.

[0069] The apparatus for acquiring three-dimensional information of objects and surfaces for an artificial vision system for the automatic optical inspection of the visual quality of an underlying object, as conceived in this way, is susceptible to numerous modifications and variants, all falling within the scope of the inventive concept as defined in the claims; moreover, all details may be replaced with technically equivalent elements.

[0070] In practice, the materials used, as well as the dimensions, may be of any type according to requirements and the state of the art.

Claims

CLAIMS1. Apparatus (1) for acquiring three-dimensional information of objects and surfaces for an artificial vision system for the automatic optical inspection of the visual quality of an underlying object (100), characterized by comprising:5 - an upper camera (10) having a vertical optical axis (Z),- a plurality of lateral cameras (20) arranged around the vertical optical axis (Z) of said upper camera (10),- a lighting system from above the object comprising a plurality of light projectors (30a, 30b, 30c, 30d) of different colors or wavelengths arranged around the vertical optical axis (Z) of said upper 0 camera (10),- an electronic controller adapted to control the simultaneous emission of light beams (32a, 32b, 32c, 32d) of different colors or wavelengths by said plurality of light projectors (30a, 30b, 30c, 30d),- each camera (10, 20) comprising spatial separation means (120) for the light beams (32'a, 32'b, 32'c, 32'd) of different colors or wavelengths reflected from the object (100) and generated by the incident light beams (32a, 32b, 32c, 32d) of different colors or wavelengths on the object (100), - and a plurality of optical sensors (110a, 110b, 110c, 1 lOd) for the simultaneous acquisition of the reflected and separated light beams (32'a, 32'b, 32'c, 32'd) of different colors or wavelengths by the separation means (120).0 2. Apparatus (1) according to claim 1, characterized in that said separation means (120) comprise at least one optical separator.

3. Apparatus (1) according to claim 2, characterized in that said at least one optical separator is formed by a multi-channel triangular optical prism, with each channel associated with a corresponding optical sensor of said plurality of optical sensors (110a, 110b, 110c, 1 lOd), which 5 in turn have their respective focal planes aligned.

4. Apparatus (1) according to claim 2, characterized in that said at least one optical separator is formed by a plurality of cubic optical prisms, each associated with a corresponding optical sensor of said plurality of optical sensors (110a, 110b, 110c, 1 lOd).

5. Apparatus (1) according to any preceding claim, characterized in that said plurality of light 5 projectors (30a, 30b, 30c, 3 Od) of different colors or wavelengths includes four light projectors of different colors or wavelengths.

6. Apparatus (1) according to any preceding claim, characterized in that each light projector (30a, 30b, 30c, 30d) comprises:- a plurality of LED light sources (30’, 30”, 30’”, 30””) of different colors selectively activatable, 0 - a lens (31),- and a Digital Micromirror Display (DMD) device (33) interposed between the lens (31) and the LED light sources (30’, 30”, 30”’, 30””).

7. Apparatus (1) according to any preceding claim, characterized by comprising a telecentric lens (55) having a vertical main optical arm (51) associated with said upper camera (10) and coaxial with said vertical optical axis (Z), and a secondary optical arm (52) branching transversely from the vertical main optical arm (51), a supplementary light projector (30e) being also associated with said secondary optical arm (52) of said telecentric lens (55).

8. Apparatus (1) according to the preceding claim, characterized in that said telecentric lens (55) is configured for acquisition by said upper camera (10) of both:0 the reflected light beams (32’ a, 32’b, 32’ c, 32’ d) generated by the light beams of different colors or wavelengths (32a, 32b, 32c, 32d) emitted by said plurality of light projectors (30a, 30b, 30c, 30d),and the reflected light beam (32’ e) generated by the light beam (32e) emitted by said supplementary light projector (30e).

9. Apparatus (1) according to the preceding claim, characterized in that said telecentric lens (55) comprises an optical prism (50) positioned at the intersection between the main optical arm (51) and the secondary optical arm (52), said optical prism (50) being configured and arranged to: direct the light beam (32e) introduced by the supplementary projector (30e) in the secondary 5 optical arm (52) into the main optical arm (51) for downward emission,and to direct the reflected light beam (32’e) returning upward in the main optical arm (51) towards the upper camera (10).

10. Apparatus (1) according to any of claims 7 to 9, characterized in that it provides at least three operating conditions:10 - a first operating condition of lighting with said plurality of projectors (30a, 30b, 30c, 30d) in combination with image acquisition by the upper camera (10);- a second operating condition of lighting with said plurality of projectors (30a, 30b, 30c, 30d) in combination with image acquisition by said plurality of lateral cameras (20);- a third operating condition of lighting with said supplementary light projector (30e) in combination 15 with image acquisition by the plurality of lateral cameras (20).0514

Images