Method and apparatus for detecting and reconstructing the 3D shape of a physical object

Abstract

According to an aspect of the disclosure a method of detecting the 3D shape of a physical object comprises receiving, by one or more computer processors (18), a series of images of the physical object (15) taken from different viewpoints relative to the object; integrating a computer-generated virtual calibration object (42) as seen from the different viewpoints with the images of the physical object to create a series of augmented-reality images (44) of the physical object with the virtual object; reconstructing a 3D model (46) from the series of augmented-reality images of the physical and virtual objects using a photogrammetric pipeline (steps (c) and (d) in Fig. 4). Another aspect of the disclosure relates to an apparatus (10) for detecting the 3D shape of a physical object.

Description

[0001] DP.LIST.0035 / PC 1

[0002] Method and apparatus for detecting and reconstructing the 3D shape of a physical object

[0003] Background

[0004]

[0001] The disclosure generally relates to a method and an apparatus for detecting and reconstructing the three-dimensional (3D) shape of a physical object. In a preferred aspect of the disclosure, the physical object includes an object with a high aspect ratio, such as, in particular, a fibre, e.g., a natural fibre.

[0005] [2] Natural fibres are widely used in industry, e.g., for the manufacturing of textiles, product consumer goods, and the manufacturing of polymer composite materials. Nevertheless, their geometrical complexity and variability pose challenges in quality and manufacturing control.

[0006] [3] Natural fibres, such as bamboo, cotton, or flax, often possess irregularities or variations in their cross-sectional geometry due to their organic nature and growth patterns. Unlike synthetic fibres, which typically have uniform cross-sections, natural fibres may feature irregularities such as voids, variations in diameter, or even noncircular shapes. During tensile testing, understanding the cross-sectional properties of natural fibres is essential for several reasons. Firstly, variations in cross-sectional geometry can significantly influence the mechanical behaviour of the fibres under tensile loading. Irregularities in the cross-section can lead to stress concentrations, affecting the distribution of mechanical forces along the fibre length and potentially resulting in premature failure or unpredictable mechanical responses. Secondly, accurate cross-sectional analysis provides insights into the structural integrity and quality of natural fibres.

[0007] [4] By examining cross-sectional features such as fibre diameter, shape irregularities, or the presence of defects, researchers can assess fibre quality, consistency, and potential performance variations. This information is vital for industries reliant on natural fibres, such as textile manufacturing, where fibre quality directly impacts the final product's durability and performance. Furthermore, precise cross-sectional analysis enables researchers to better understand the structureproperty relationships of natural fibres. By correlating cross-sectional characteristics with mechanical properties such as tensile strength, modulus, and elongation, DP.LIST.0035 / PC 2 scientists can elucidate how fibre morphology influences mechanical behaviour. This knowledge aids in the development of optimized fibre processing techniques, composite materials, and engineering applications involving natural fibres.

[0008] [5] US 9990767 discloses a system and methods for generating 3D models using imaging data obtained from an array of camera devices. A stochastic shape distribution is applied upon an object to be modelled in invisible ink. The system activates a first lighting mode which causes the stochastic shape distribution to be visible and captures a first set of images that depict the stochastic shape distribution on the object. The system then activates a second lighting mode that causes the stochastic shape distribution to be hidden and captures images of the object as it would normally appear (without the stochastic shape distribution). The images having the stochastic shape distribution are used to determine alignment information for the images within the set of images. That alignment information may then be attributed to the second set of images and used to generate the 3D model.

[0009] [6] WO 2020169853 relates to a microscale optical capture system, which has little divergence of incident light and a low acceptance angle of captured light. The microscale optical system has a large number of high-power collimated white LEDs as light sources, which can be placed at distances of approximately 650 mm from a sample. A digital camera that uses a 50-mm focal lens with a 25-mm long extension tube captures images of the sample. This provides a working distance of approximately 100 mm, and at the same time, maintains a magnification of more than 0.5x for microscale captures, with an image size of, for example, 4x4 microns per pixel.

[0010] [7] US2008101688A1 uses a structured light pattern digitizing method is combined with photogrammetry to determine a 3D model of an object. The structured light digitizing operation generates a 3D model of the object being scanned, and this model is then used to compute a higher accuracy model using photogrammetry.

[0011] [8] CN1 16399314 relates to a calibration device for photogrammetry. The calibration device mainly comprises a calibration ball, a plurality of equilateral patterns are uniformly distributed on the surface of the calibration ball, and the colour of the equilateral graphs is different from the background colour of the calibration ball. According to the document, the calibration ball facilitates three-dimensional calibration and improves measurement accuracy. Also, the solving precision of image features DP.LIST.0035 / PC 3 and multi-view geometric constraints can be improved, so that the precision of the 3D reconstruction model is effectively corrected.

[0012] [9] The teachings of the above documents were found not to satisfactorily address the problems encountered in detecting the 3D shape of small objects, in particular small objects with high aspect ratios, such as fibres.

[0013]

[0010] The object of the present disclosure has arisen from the need to address these challenges but has more general applicability. In particular, aspects of the disclosure may be applicable to photogrammetry of larger-scale objects. Nevertheless, preferred aspects of the disclosure relate to photogrammetry of small objects with high aspect ratio, such as, e.g., fibres, in particular natural fibres.

[0014] Summary

[0015]

[0011] In a first aspect, the disclosure relates to a method of detecting the 3D shape of a physical object. The method comprises receiving, by one or more computer processors, a series of images (e.g., optical microscopy images) of the physical object taken from different viewpoints relative to the object; integrating a computer-generated virtual calibration object as seen from the different viewpoints with the images of the physical object to create a series of augmented- reality images of the physical object with the virtual object; reconstructing a 3D model from the series of augmented-reality images of the physical and virtual objects using a photogrammetric pipeline.

[0016]

[0012] The one or more computer processors may include a general-purpose microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field- programmable gate array (FPGA), a programmable logic controller (PLC) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The one or more computer processors may also comprise a combination of computing devices, for example, a combination of one or more CPUs and one or more GPUs, a plurality of microprocessors, multicore processors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. DP.LIST.0035 / PC 4

[0017]

[0013] As used herein, the expression “photogrammetric pipeline” designates a (partly or fully computer-implemented) workflow that takes as input a set of images of an object under different angles and that produces a 3D model of the object as output. The photogrammetric pipeline could be implemented using software such as, e.g., RealityCapture, Metashape, COLPMAP, OpenMVG, Meshroom, etc.

[0018]

[0014] The expression “3D model” designates a representation of an object and in the form of a collection of data (e.g. a computer file or a data record), including points (sets of coordinates) in 3D space, connected by geometric entities, such as, e.g., triangles, lines, curved elementary surfaces, etc., to define the surface(s) of the object and its volume(s). The surface(s) may include texture(s) and other attributes.

[0019]

[0015] The virtual object may be seamlessly integrated into the imaging process, perfectly synchronized with the motion of the physical object being studied, thereby facilitating precise reconstruction of the actual object.

[0020]

[0016] It has been found that, in particular at the microscale, the use of virtual objects as a reference for photogrammetry is beneficial due to the challenges posed by illumination and the limited depth of field of microscope lenses. When dealing with objects at the microscale, such as fibres or other small objects with a high aspect ratio, it becomes increasingly difficult to maintain proper focus on the entire object during scanning. On the one hand, the optical imaging systems of the cameras employed for imaging the physical object may necessitate microscope objectives to generate pictures with sufficiently high resolution to enable meaningful 3D reconstruction. On the other hand, the narrow depth of field of microscope objectives makes it nearly impossible to keep the entire fibre in focus at a time. Furthermore, it may be impossible, in practical applications, to focus a background (object) for reference at the same time as the object of interest. By incorporating a virtual calibration object (e.g., as a background element) into the photogrammetric setup, the illumination (including the illumination of the background) can be precisely controlled and optimized for accurate reconstruction. The virtual object can be rendered with consistent lighting conditions, ensuring that the captured images have uniform illumination across the entire field of view. This is particularly interesting for small objects like fibres, where the high aspect ratio can lead to uneven lighting and shadows if a real calibration object is used. Moreover, the virtual object can be designed to provide one or more stable reference planes for the photogrammetric reconstruction. Since the depth of field of microscope DP.LIST.0035 / PC 5 lenses is limited, using a real calibration object could result in parts of the physical object of interest being out of focus, leading to inaccuracies in the reconstruction. By employing a virtual calibration object, the entire calibration object can be kept in sharp focus, providing a reliable reference for the photogrammetric algorithms to align the images and reconstruct the geometry of the physical object accurately.

[0021]

[0017] It should be noted that virtual calibration objects could also be used in the photogrammetric reconstruction of macroscopic objects. However, the technique using the virtual calibration object is considered as a groundbreaking facilitator of optical photogrammetry of microscale objects.

[0022]

[0018] Preferably, integrating the virtual object as seen from the different viewpoints with the images of the physical object comprises registering the virtual object with the physical object and / or the different viewpoints. The virtual object could be incorporated into the images of the physical object as a background. Registering the virtual object with the physical object and / or the different viewpoints could be carried out in two or more steps, including an initial rough positioning step and one or more fine positioning steps. An iterative closest point (ICP) algorithm could be used in this step.

[0023]

[0019] According to embodiments, reconstructing the 3D model includes generating a dense point cloud, which may then be used to generate a mesh therefrom. Prior to generating the dense point cloud, the method may include the generation of a sparse point cloud.

[0024]

[0020] Furthermore, reconstructing the 3D model may include texture mapping and / or illumination adjustment.

[0025]

[0021] The virtual calibration object may comprise one or more reference marks, e.g., specific geometric patterns (e.g. circles, polygons, etc.) in known positions on the calibration object.

[0026]

[0022] The series of images of the physical object may include frames from video footage. According to embodiments, the series of images of the physical object may be recorded while the physical object is rotated (about one or more axes of rotation), and, optionally, translated, relative to the one or more cameras.

[0027]

[0023] According to preferred embodiments, the one or more cameras may include a long-distance microscope having a working distance (distance between the object and DP.LIST.0035 / PC 6 the objective) of at least 10 mm, preferably of at least 13 mm, more preferably of at least 20 mm and even more preferably of at least 30 mm.

[0028]

[0024] Preferably, relative position and / or motion data between the physical object and the one or more cameras are logged during the recording of the series of image. The relative position and / or motion data may then be used to integrate the computer- generated virtual calibration object as seen from the different viewpoints with the images of the physical object. Logging relative position and / or motion data between the physical object and the one or more cameras might not be necessary, especially if the relative motion between the physical object and the one or more cameras is known or can be computed from respective trajectory and orientation data. When the physical object and / or the one or more cameras are mounted on motorized kinematic mounts controlled to carry out preprogrammed translatory and / or rotary motions (in a common reference system) the logging operation may complement the data of the preprogrammed motions or may be omitted.

[0029]

[0025] In a preferred application, the physical object may comprise a fibre or a fibre bundle, e.g., a natural fibre (bundle), a synthetic fibre (bundle), or a composite fibre (bundle). Specifically, the physical object may comprise a natural fibre (bundle), for instance a bamboo fibre or a bamboo fibre bundle. Other examples of usable natural fibres include flax fibres, cotton fibres, wool fibres, hemp fibres, etc.

[0030]

[0026] The series of images of the physical object, in particular, a fibre or a fibre bundle, may be recorded with one or more cameras while the physical object is rotated with respect to the one or more cameras. The physical object may be positioned in the field of view of the one or more cameras with a fibre positioning system comprising a first hollow (micro-)cylinder and a second hollow (micro-)cylinder, the first and second hollow cylinders being arranged coaxially on the axis of rotation and separated by a gap at least partially within the field of view of the one or more cameras. Preferably, the cylinders have an internal diameter slightly larger than the physical object under investigation, e.g., between 50 pm and 1 mm (depending on the diameter of the fibre or fibre bundle).

[0031]

[0027] The physical object may be moved axially through the fibre positioning system continuously or intermittently. The series of images of the physical object may be recorded with the one or more cameras while the physical object is axially moved or is axially at rest. DP.LIST.0035 / PC 7

[0032]

[0028] Relative motion between the physical object and the one or more cameras may be achieved by moving the one or more cameras, the physical object, or both. According to embodiments, the one or more cameras may be mounted on a rotary stage, preferably a motorized rotary stage, capable of rotating about the axis of rotation. Alternatively, or additionally, the physical object may be guided on a rotary device configured for rotating the physical object at least in the field of view of the one or more cameras.

[0033]

[0029] A preferred aspect of the disclosure relates to an apparatus for detecting the 3D shape of a physical object, comprising: one or more cameras for recording images of a physical object; an object stage for positioning the physical object in a field of view of the one or more cameras; a rotary stage carrying the one or more cameras for changing the viewpoints of the one or more cameras relative to the object and / or a rotary device for rotating the physical object at least in the field of view of the one or more cameras; one or more computer processors operatively connected to the one or more cameras for receiving a series of images of the physical object taken from different viewpoints relative to the object, the one or more computer processors configured for integrating a computer-generated virtual calibration object as seen from the different viewpoints with the images of the physical object to create a series of augmented-reality images of the physical object with the virtual object and for reconstructing a 3D model from the series of augmented-reality images of the physical and virtual objects using a photogrammetric pipeline.

[0034]

[0030] The one or more cameras may include a long-distance microscope having a working distance of at least 10 mm, preferably of at least 13 mm, more preferably of at least 20 mm and even more preferably of at least 30 mm.

[0035]

[0031] The apparatus may preferably comprise a data logger for logging relative position and / or motion data between the physical object and the one or more cameras during the recording of the images of the physical object. The data logger may be operatively connected to one or more computer processors. The one or more computer processors may be configured to use the relative position and / or motion data to DP.LIST.0035 / PC 8 integrate the computer-generated virtual calibration object as seen from the different viewpoints with the images of the physical object.

[0036]

[0032] The apparatus may also include a fibre positioning system comprising a first hollow cylinder and a second hollow cylinder, the first and second hollow cylinders being arranged coaxially on an axis of relative rotation between the one or more cameras the physical object, the first and second hollow cylinders being separated by a gap that is at least partially within the field of view of the one or more cameras.

[0037]

[0033] It will be appreciated that the method / apparatus according to the disclosure may by applied for examining fibres, in particular natural fibres. Thanks to the method / apparatus, geometrical features, such as cross-section shape, fibre diameter, shape irregularities, defects, etc. can be studied on the generated true-to-scale 3D model. The geometrical data can be cross-correlated with measured mechanical properties (e.g., tensile strength, modulus, elongation, etc.), which allows gaining a better understanding of how fibre morphology influences mechanical behaviour. The present method / apparatus could be used for the systematic examination of continuous fibres. In such a method, the fibre could be scanned segment by segment (with some overlap between neighbouring segments), enabling a 3D reconstruction of fibres of virtually any length. Images of one or more segments could be input to the photogrammetric pipeline at a time to reconstruct a 3D model of these segments. 3D models of different segments or sets of segments could thereafter be put together like pieces of a puzzle, e.g., using axial overlaps for seamless integration.

[0038]

[0034] In the present document, the verb “to comprise” and the expression “to be comprised of’ are used as open transitional phrases meaning “to include” or “to consist at least of”, not excluding the presence of further features or components. Unless otherwise implied by context, the use of singular word form is intended to encompass the plural, except when the cardinal number “one” is used: “one” herein means “exactly one”. Ordinal numbers (“first”, “second”, etc.) are used herein to differentiate between different instances of a generic object; no particular order, importance or hierarchy is intended to be implied by the use of these expressions. Furthermore, when plural instances of an object are referred to by ordinal numbers, this does not necessarily mean that no other instances of that object are present (unless this follows clearly from context). When this description refers to “an embodiment”, “one embodiment”, “embodiments”, etc., this means that the features of those embodiments can be used DP.LIST.0035 / PC 9 in the combination explicitly presented but also that the features can be combined across embodiments without departing from the disclosure, unless it follows from context that features cannot be combined.

[0039] Brief Description of the Drawings

[0040]

[0035] By way of example, preferred, non-limiting embodiments of the disclosure will now be described in detail with reference to the accompanying drawings, in which:

[0041] Fig. 1 : is a schematic illustration of an apparatus for detecting the 3D shape of a physical object;

[0042] Fig. 2: is a perspective view of a detail of the apparatus of Fig. 1 ;

[0043] Fig. 3: is an exploded view of the camera system of the apparatus of Fig. 1 ;

[0044] Fig. 4: is an illustration of a method for detecting the 3D shape of a physical object according to an embodiment of the disclosure.

[0045] Detailed Description of a Preferred Embodiment

[0046]

[0036] According to a preferred embodiment, it is proposed to utilize a virtual calibration object as a reference to enhance 3D modelling at the microscale through micro photogrammetry. To the inventors’ best knowledge, no current equipment using optical microscopy is capable of performing micro photogrammetry at this scale, making the technique disclosed herein truly groundbreaking. In the preferred embodiment, the method involves several steps, including capturing images of the physical object with a long-distance microscope and seamlessly integrating the virtual calibration object into the images for improved photogrammetric reconstruction accuracy thanks to the thus obtained augmented-reality images.

[0047]

[0037] Micro-photogrammetry is a computer-implemented technique for the reconstruction of 3D models of small objects, such as, e.g., fibres, from overlapping images of the objects from multiple angles. Incorporating a virtual calibration object into the images that are input into the photogrammetric pipeline address challenges related to illumination and depth of field, leading to more precise reconstructions. The complete process preferably includes image capture, virtual calibration object generation, registration of the virtual object with the physical object, e.g., using geometric transformation matrices, and the use of photogrammetric software for the DP.LIST.0035 / PC 10 generation of the 3D model. It was found that the use of a virtual calibration object may significantly enhance the accuracy of 3D modelling at the microscale.

[0048]

[0038] When micro-scale objects are to be imaged, the lens-camera configuration needs to be carefully balanced to obtain the desired depth of field. Preferably, to observe natural fibres or objects of similar diameters, the depth of field is at least 10 pm. Controlled motion steps and stable support systems for both the camera(s) and the physical object are required to ensure sharp pictures without any disruptions during the image capture process.

[0049]

[0039] To visualize an object of interest volumetrically, the optical system of the camera(s) needs to guide sufficient light from the object onto the camera sensor. So, an appropriate depth of field for the object size depends on the numerical aperture (NA) of the optical system.

[0050]

[0040] An example of an apparatus 10 for detecting the 3D shape of a natural fibre 15 is generally shown in Fig. 1. The apparatus 10 comprises a camera system 12, an object stage 14, a drive controller 16, and a computer 18. A detail of the object stage is shown in Fig. 2.

[0051]

[0041] The camera system 12 comprises a digital camera 20 (e.g. a sCMOS camera with a 2 / 3” sensor with 2448x2048 pixels at a frame rate of up to 70 fps) coupled with an optical system 22 including long-distance magnification lenses for capturing images or videos of the sample (physical object) according to user-defined focus and field of view. The camera 20 and the optical system 22 are fixed on a motion system 24, which includes, for example, a manual lab jack for coarse adjustment of the height of the camera system, a semiautomatic 3-axis (x, y, z) linear stage, and a manual rotary stage. The manual rotary stage allows adjusting the inclination of the camera and the optical system.

[0052]

[0042] Fig. 3 shows an exploded view of an example of the camera system 12. Using an sCMOS camera PRT-001 with a sensor size of 2 / 3”, a C-mount PRT-002 is utilized as a coupling element with the optical accessories assembled. A 1x adjustable adapter PRT-003 is used for telescoping the optics and it is coupled with 12x zoom lenses PRT-004 and to a converter lens PRT-005. In this configuration the zoom lenses PRT- 004 can vary the magnification between 0.58x to 7x using the 1x adapter allowing a dept of field between 1 .39-0.05 mm, with a working distance of 86 mm. In this case the DP.LIST.0035 / PC 11 field of view is 18.97 mm and 1.75 mm, for low and high magnification values, respectively.

[0053]

[0043] When the converter lens PRT-005 is replaced by the coupler PRT-006 assembled with a 10x objective lens PRT-007, a 12x-ultra-zoom configuration is achieved. The wording distance in this case is 33 mm.

[0054]

[0044] The object stage 14 is configured for placing the fibre 15 to be monitored. The object stage 14 comprises a needle alignment system 28 comprising a first hollow cylinder 29A mounted on a first positioning stage 30 and a second hollow cylinder 29B mounted on a second positioning stage 32. The first and second positioning stages 30, 32 are configured such that the first and second hollow cylinders 29A, 29B can be brought precisely into axial alignment with each other with a gap between them. In the illustrated embodiment, the first positioning stage 30 comprises a precision fibre alignment 3-axis positioner stage 34 (commercially available for the alignment of optical fibres) and a first holder 36 carrying the first hollow cylinder 29A. The second positioning stage 32 comprises a precision fibre alignment 2-axis positioner stage carrying a motorized rotary stage 38. A second holder 40 placed on the rotary stage 38 carries the second hollow cylinder 29B. The fibre 15 is placed in the hollow cylinders 29A, 29B to bridge the gap between them and secured against rotation with respect to the second hollow cylinder 29B, e.g. by means of one or more transversal screws, a clamp or another fixture. The rotary stage 38 serves to rotate the fibre 15 during the picture-taking process. The fibre 15 may freely rotate about its axis within the first hollow cylinder 29A, the function of which is to maintain the fibre aligned during the rotation.

[0055]

[0045] The object stage 14 may further comprise lights to illuminate the object during rotation and image-recording.

[0056]

[0046] In the illustrated embodiment, vertical alignment of each hollow cylinder and thus collinearity of the first and the second hollow cylinders is achieved by design of the supports. Accordingly, the degrees of freedom of the first and second positioning stages 30, 32 need only be translatory, in order to achieve co-axiality and the desired distance between the first and second hollow cylinders. However, in order to correct the collinearity of the first and the second hollow cylinders, one or both of the positioning stages 30, 32 could be equipped with an additional tip-tilt stage. DP.LIST.0035 / PC 12

[0057]

[0047] The different motorized components of the camera system 12 and the object stage 14 are connected to the drive controller 16. The drive controller 16 is connected to a joystick 26 and to computer 18. The motorized stages may be moved semiautomatically with the joystick 26, e.g., to position the camera system 12 relative to the object under investigation, or automatically under the control of the computer 18 running appropriate software. The drive controller 16 is preferably configured to log data relating to the motion and / or position of the motorized stages and to make these data available to the computer 18.

[0058]

[0048] The computer 18 is connected to the digital camera 20 so that the images taken by the camera may be transferred to the computer 18.

[0059]

[0049] It may be worthwhile noting that, although the data links between the have been shown in the drawing as wire-based, one or more of these data links could be implemented using wireless communication protocols if the respective individual components are equipped with corresponding wireless interfaces.

[0060]

[0050] Fig. 4 is an illustration of a method of detecting the 3D shape of a fibre according to an embodiment of the disclosure.

[0061]

[0051] In a first step, step (a), also referred to as the scanning process, images of the fibre taken are from different viewpoints relative to the fibre. In the setup of Figs. 1 and 2, this means that images of the fibre are recorded by the camera system 12 while the fibre is rotated, e.g., at a constant angular speed. In alternative embodiments, the fibre (or the physical object) remains immobile, while the one or more cameras are moved around the fibre. In yet further embodiments, both the object and the one or more cameras could be moved during the taking of images. As in conventional photogrammetry there has to be significant overlap between the images, so that features appear in different images and under different angles.

[0062]

[0052] The images are transmitted to the computer 18, which further generates views of a virtual calibration object (e.g., a cube 42) as seen from the different viewpoints. It is worthwhile noting that the initial position of the virtual calibration object may be chosen arbitrarily. However, the further views of the virtual calibration objects are generated by simulating the same relative movements between the viewpoint and the virtual object than between the one or more cameras and the physical object. In other words, for each image of the physical object taken from a specific point of view, the DP.LIST.0035 / PC 13 computer generates an image of the virtual calibration object from the same point of view.

[0063]

[0053] The virtual calibration object may comprise one or more reference marks (e.g., dot patterns, geometric patterns, etc.) that can easily be identified. The reference marks may comprise (virtual) dot patterns that are strategically designed with precise geometric arrangements and known dimensions, enabling the calibration process to adjust for scale, alignment, and orientation throughout the 3D reconstruction. The reference marks constitute highly reliable reference points that enhances the photogrammetric process by ensuring consistency and reducing reconstruction errors. The placement of the reference marks on the virtual calibration object is preferably such that at least part of the marks is present in every frame.

[0064]

[0054] The virtual calibration object can be generated using 3D modelling software (e.g., SolidWorks, AutoCAD, etc.) The views of the virtual calibration object can then be obtained using a motion study scene of a CAD software. The motion of the virtual object follows the motion of the physical object relative to the one or more cameras. If the physical object is rotated about a given axis of rotation in the field of view of the camera(s) taking pictures at a fixed frame rate, the virtual object is rotated at the same angular velocity in the virtual studio and digital images of the virtual object are acquired (e.g., with a black background) at the same frame rate.

[0065]

[0055] To synchronize the motion of the physical object with the motion of the virtual calibration object, the computer 18 may use logged position and / or motion data of the physical object and / or the one or more cameras. It may be worthwhile noting that there may be no need to log position and / or motion data of the physical object and / or the one or more cameras, e.g., when the relative motion between the physical object and the one or more cameras is known or can be computed from respective trajectory and orientation data. This may be the case when the physical object and / or the one or more cameras carry out preprogrammed motions in a common reference system. When the motion or position data are sufficiently accurate, registering the virtual object with the physical object and / or the different viewpoints may be carried out in a single step. However, when the motion or position data not are sufficiently accurate, registering the virtual object with the physical object and / or the different viewpoints may be carried out in at least two steps, including an initial rough positioning step that relies on the motion and / or position data and one or more fine positioning steps wherein the geometrical DP.LIST.0035 / PC 14 transformation matrices (that describe the relative motion from one image to the next) are corrected based upon the views of the physical object, e.g., using an iterative closest point (ICP) algorithm.

[0066]

[0056] The computer 18 then integrates the views of the virtual calibration object as seen from the different viewpoints with the corresponding images of the physical object, so as to create a series of augmented-reality images 44 of the physical object with the virtual calibration object (step (b) in Fig. 4).

[0067]

[0057] These augmented-reality images of the physical object with the virtual calibration object are then input to a photogrammetric pipeline for reconstructing a 3D model 46. The photogrammetric pipeline could be implemented using software such as, e.g., RealityCapture, Metashape, COLPMAP, OpenMVG, Meshroom, etc. The computer 18 may be configured to reconstruct the 3D model 46 automatically, with or without prompting the user to adapt parameters of the software. The individual steps of the photogrammetric pipeline need not be detailed herein. Preferably, the reconstruction of the 3D model includes generating a dense point cloud 48 (step (c) in Fig. 4), and generation of a mesh 50 from the dense point cloud (step (d) in Fig. 4). The mesh may be output or represented in the STL file format or any other suitable file format for representing 3D meshes. In one or more final steps, the mesh may be converted into a CAD model, e.g., by adding colour, texture, scale and / or other CAD model attributes. Texture may be applied by texture mapping using the original images of the physical object and the views of the virtual calibration object. Finally, the 3D model may be rendered with consistent lighting, e.g., matching the lighting of the physical setup.

[0068]

[0058] As will be appreciated, the virtual calibration object may operate as control volume to scale the images of the of the physical object.

[0069]

[0059] During the scanning process, the physical object may be rotated at constant angular velocity. For focusing the physical object, the optical system of the camera system may be equipped with long-distance magnification lenses that are set up to the required focus according to the shape of the object and to a depth of field equal to or greater than the thickness of the object to avoid blurred images. Initially, the fibre may be mounted and aligned on the object stage of the device to ensure optimal positioning. DP.LIST.0035 / PC 15

[0070]

[0060] Preferably, the computer-implemented integration of the virtual calibration object may be carried out in real time, so that the operator can visualise the augmented-reality images during the scanning process.

[0071]

[0061] The integration of a virtual calibration object helps to resolve illumination and focus issues during reconstruction. Since the virtual object is seamlessly integrated into the scanning process, synchronized with the relative motion of the physical object under investigation, precise reconstruction of the 3D model of the physical object (as a part of the 3D model of the augmented-reality object) is facilitated. In particular, that virtual calibration object provides a stable reference frame (which survives, in particular, the step of mesh generation, where scale information may be lost).

[0072]

[0062] At the microscale, employing virtual objects for photogrammetry resolves challenges posed by the limited depth of field in microscope lenses. In particular, manipulating physical calibration objects and focussing both the object of interest and the physical calibration object are no longer necessary. When working with high aspect ratio objects, such as fibres, maintaining focus across the entire object during scanning becomes increasingly challenging. The narrow depth of field typical of microscope objectives often results in only a small portion of the object being in focus at any given time. By incorporating a virtual calibration object, the depth of field of the camera system(s) can be optimized by focusing solely on the object of interest, as there is no need to image any physical calibration object.

[0073]

[0063] The need for physical calibration objects and the problems relating to their use (focussing, illumination, shadows, etc.) are believed to be the key reasons why photogrammetry has not yet been widely implemented for optical microscale imaging.

[0074]

[0064] Because the calibration object used in the context of the disclosure is a virtual object, constraints relating to depth of field and to illumination of a physical background and / or a physical calibration object are also avoided. During the scanning process, illumination conditions can thus be controlled much better and under less constraints, ensuring accurate reconstruction. The virtual object can indeed be rendered under consistent lighting, so that uniform illumination of the augmented-reality object can be achieved across the entire field of view in each image. This is particularly interesting for materials like fibres, where high aspect ratios can lead to uneven lighting and shadows if a real background is used. Furthermore, the virtual background can be designed to provide a stable reference plane for photogrammetric reconstruction. DP.LIST.0035 / PC 16

[0075] Since real objects can introduce additional focus issues due to their physical properties, using a virtual object allows us to maintain sharp focus across the entire background, enhancing the reliability of photogrammetric algorithms in aligning images and accurately reconstructing fibre geometry.

[0076]

[0065] Integrating customizable virtual calibration objects into the 3D model reconstruction workflow offers a powerful approach for improving accuracy and flexibility in scientific measurement systems. In the above-discussed method, a virtual calibration object (which may be designed with a customizable (dot) pattern) is superimposed with video footage of the physical object under investigation.

[0077]

[0066] The virtual calibration object can be tailored to the specific needs of each 3D model reconstruction project. The size and the shape of the virtual object itself but also the dimensions, patterns, etc. of the reference marks on the virtual object may be adapted to the requirements of each experiment or measurement task. The virtual calibration object may be programmed to move or rotate synchronously with the physical object providing continuous calibration possibility throughout the video. The virtual calibration object ensures that key metrics, such as distances and angles, are maintained correctly, offering more accurate 3D models. By combining virtual calibration objects with video-based photogrammetry, users can eliminate the need for physical calibration objects, reduce setup complexity, and ensure the seamless integration of virtual and physical data. This leads to higher precision in reconstructing complex objects and opens new possibilities in fields like microscopy, material science, and industrial metrology.

[0078]

[0067] According to a preferred embodiment, a 3D scanning plugin is provided. The 3D scanning plugin allows for reconstructing complex shapes from videos and augments the scope of microscopic scientific instrumentation. Frames from a video of the physical object may be combined with frames of a computer-generated calibration video (showing the virtual calibration object), allowing for accurate scale and alignment. All or part of the frames of the augmented video may then be used as input of a photogrammetry pipeline. The plugin leverages photogrammetric reconstruction which extracts 3D information from 2D images. The use of the virtual calibration object helps to ensure accurate scaling and positioning in the 3D reconstruction. The plugin may be configured to compute a dense point cloud and / or a 3D mesh as its output. A CAD re-modelling tool can then be used for surface repair and reconstruction. DP.LIST.0035 / PC 17

[0079]

[0068] It may be worthwhile noting that in the 3D model (or any intermediary data sets such as the dense point cloud or the mesh), features belonging to the virtual calibration object can at any time be identified as such. Accordingly, after computation of the 3D model of the augmented-reality object, it is possible to extract a true-to-scale 3D model of the physical object. The virtual calibration object can remain part of the dataset or be removed from it. Treating the 3D model of the physical object and the virtual calibration object as separate entities in the CAD software e.g. allows choosing different properties for them. For instance, the user could change the way the virtual calibration object is displayed. The virtual calibration object could, e.g., be displayed as a semi-transparent object while the 3D model of the physical object is displayed without any transparency.

[0080]

[0069] While specific embodiments and examples have been described herein in detail, those skilled in the art will appreciate that various modifications and alternatives to those details could be developed in light of the overall teachings of the disclosure. Accordingly, the particular arrangements disclosed are meant to be illustrative only and not limiting as to the scope of the disclosure, which is to be given the full breadth of the appended claims and any and all equivalents thereof.

Claims

DP.LIST.0035 / PC 18Claims1 . A method of detecting the 3D shape of a physical object, comprising: receiving, by one or more computer processors, a series of images of the physical object taken from different viewpoints relative to the object; integrating a computer-generated virtual calibration object as seen from the different viewpoints with the images of the physical object to create a series of augmented-reality images of the physical object with the virtual object; reconstructing a 3D model from the series of augmented-reality images of the physical and virtual objects using a photogrammetric pipeline.

2. The method as claimed in claim 1 , wherein integrating the virtual object as seen from the different viewpoints with the images of the physical object comprises registering the virtual object with the physical object and / or the different viewpoints.

3. The method as claimed in claim 2, wherein registering the virtual object with the physical object and / or the different viewpoints is carried out in at least two steps, including an initial rough positioning step and one or more fine positioning steps.

4. The method as claimed in any one of claims 1 to 3, wherein reconstructing the 3D model includes generating a dense point cloud.

5. The method as claimed in claim 4, wherein reconstructing the 3D model includes generating a mesh from the dense point cloud.

6. The method as claimed in claim 5, wherein reconstructing the 3D model includes texture mapping.

7. The method as claimed in any one of claims 1 to 6, wherein the virtual calibration object comprises one or more reference marks.

8. The method as claimed in any one of claims 1 to 7, wherein the series of images of the physical object include frames from video footage.

9. The method as claimed in any one of claims 1 to 8, wherein the series of images of the physical object are recorded with one or more cameras while the physical object is rotated with respect to the one or more cameras at least in the field of view of the one or more cameras.DP.LIST.0035 / PC 1910. The method as claimed in claim 9, wherein the one or more cameras include a long-distance microscope having a working distance (distance between the object and the objective) of at least 10 mm, preferably of at least 13 mm.

11. The method as claimed in any one of claims 1 to 10, wherein relative position and / or motion data between the physical object and the one or more cameras are logged during the recording of the series of images and wherein the relative position and / or motion data are used to integrate the computer-generated virtual calibration object as seen from the different viewpoints with the images of the physical object.

12. The method as claimed in any one of claims 1 to 11 , wherein the physical object comprises a fibre or a fibre bundle, e.g., a natural fibre (bundle), a synthetic fibre (bundle), or a composite fibre (bundle).

13. The method as claimed in claim 12, wherein the physical object comprises a bamboo fibre or a bamboo fibre bundle.

14. The method as claimed in claim 12 or 13, wherein the series of images of the physical object are recorded with one or more cameras while the physical object is rotated with respect to the one or more cameras, wherein the physical object is positioned in the field of view of the one or more cameras with a fibre positioning system comprising a first hollow cylinder and a second hollow cylinder, the first and second hollow cylinders being arranged coaxially on the axis of rotation and separated by a gap at least partially within the field of view of the one or more cameras.

15. The method as claimed in claim 14, wherein the physical object is moved axially through the fibre positioning system continuously or intermittently, and wherein the series of images of the physical object are recorded with the one or more cameras while the physical object is axially moved or is axially at rest.

16. The method as claimed in claim 13 or 14, wherein the one or more cameras are mounted on a rotary stage, preferably a motorized rotary stage, capable of rotating about the axis of rotation.

17. The method as claimed in any one of claims 13 to 15, wherein the physical object is guided on a rotary device configured for rotating the physical object at least in the field of view of the one or more cameras.DP.LIST.0035 / PC 2018. An apparatus for detecting the 3D shape of a physical object, comprising: one or more cameras for recording images of a physical object; an object stage for positioning the physical object in a field of view of the one or more cameras; a rotary stage carrying the one or more cameras for changing the viewpoints of the one or more cameras relative to the object and / or a rotary device for rotating the physical object at least in the field of view of the one or more cameras; one or more computer processors operatively connected to the one or more cameras for receiving a series of images of the physical object taken from different viewpoints relative to the object, the one or more computer processors configured for integrating a computer-generated virtual calibration object as seen from the different viewpoints with the images of the physical object to create a series of augmented-reality images of the physical object with the virtual object and for reconstructing a 3D model from the series of augmented-reality images of the physical and virtual objects using a photogrammetric pipeline.

19. The apparatus as claimed in claim 18, wherein the one or more cameras include a long-distance microscope having a working distance of at least 10 mm, preferably of at least 13 mm.

20. The apparatus as claimed in claim 18 or 19, comprising a data logger for logging relative position and / or motion data between the physical object and the one or more cameras during the recording of images of the physical object, the data logger being operatively connected to one or more computer processors and one or more computer processors being configured to use the relative position and / or motion data to integrate the computer-generated virtual calibration object as seen from the different viewpoints with the images of the physical object.

21. The apparatus as claimed in any one of claims 18 to 20, including a fibre positioning system comprising a first hollow cylinder and a second hollow cylinder, the first and second hollow cylinders being arranged coaxially on an axis of relative rotation between the one or more cameras the physical object, the first and second hollow cylinders being separated by a gap that is at least partially within the field of view of the one or more cameras.

Images