Method for characterising microscopic objects by holography

A nano-computer-based prediction model processes holograms to generate phase and amplitude images and segmentation masks in real-time, addressing the inefficiencies of sequential holographic characterization methods by using a database of simulated holograms trained through machine learning.

WO2026233874A1PCT designated stage Publication Date: 2026-11-12UNIVERSITE DE NOUVELLE CALEDONIE +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2026/062687
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-03
Filing Date
2026-05-02
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Current holographic characterization methods are time-consuming and require significant computational resources, as they perform tasks sequentially and lack efficient algorithms for real-time processing, especially for tasks like phase and amplitude reconstruction, segmentation, and classification of microscopic objects, particularly in dynamic environments.

Method used

A method using a nano-computer-based prediction model that simultaneously processes raw holograms to generate phase and amplitude images, segmentation masks, and object classification, independent of focus distance, by leveraging a database of simulated holograms trained through machine learning.

Benefits of technology

Enables real-time characterization of multiple microscopic objects with reduced computational resources, providing accurate phase and amplitude images, segmentation masks, and classification, even in dynamic conditions, without requiring sequential processing or prior focus adjustment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2026062687_12112026_PF_FP_ABST
    Figure EP2026062687_12112026_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed is a method for characterising by digital holography several microscopic objects in a sample to be studied, comprising a step of capturing an image of the sample with a view to extracting therefrom a raw hologram of the sample and a step of automatically implementing a prediction model in order to catalogue each of the objects of the sample and deduce therefrom in real time for each object a bounding box bounding each object, a reconstructed phase image, a reconstructed amplitude image and a segmentation mask of each object, as well as the name assigned to the object, and the position of the object in the sample. The prediction model is trained beforehand by machine learning using simulated holograms of samples constructed from raw holograms. The method also makes it possible to increase in the database the number of images of rare objects, with a view to balancing in terms of number the database of the objects captured in the sample.
Need to check novelty before this filing date? Find Prior Art

Description

Method for characterizing microscopic objects by holography

[0001] The invention relates to the field of optical microscopy by digital holography, in particular for identifying small objects, typically millimeter-sized or microscopic, fixed or moving including in a fluidic medium.

[0002] The invention relates to the characterization of a multitude of objects found in a sample. The invention will be described more particularly with regard to an application involving biological objects, but is not limited to this. In particular, the size of the observable objects is between 5 micrometers and 1 mm.

[0003] Holography reconstructs a three-dimensional image of an illuminated object photographed on a two-dimensional surface. Biological objects are often not flat, and holography, by reconstructing a three-dimensional image, allows access to the size, shape, and spatial position of the object under study. Furthermore, holography offers the advantage of observing microorganisms not only without damaging them but also in real time, in situ.

[0004] Current holographic characterization algorithms each perform a distinct task. Some algorithms calculate phase and amplitude to create a holographic reconstruction, others determine the object's location, particularly by tracing bounding boxes, while still others perform segmentation (dividing the image into pixels that allow for precise delimitation of shapes and areas occupied by different objects). Still others can classify objects into categories. Currently, each task is performed sequentially, which is time-consuming, and the algorithms still have drawbacks, such as those described below.

[0005] Indeed, holographic reconstruction requires iterative steps (several dozen iterations) based on Fourier transform calculations that are very memory-intensive and do nothing to accelerate the processing. These methods are therefore incompatible with real-time processing and cannot be implemented by a system with low computational resources and a compact size suitable for integration into the imaging device.

[0006] A method for detecting and classifying objects such as plankton is known from prior art, notably from the paper by Romane Scherrer et al., "Real-Time Automatic Plankton Detection, Tracking and Classification on Raw Hologram." This method proposes using a deep learning-based object detection model (a YOLOv5-type convolutional neural network) specifically trained on a database of simulated holograms. However, this method is limited to localization (using a bounding box) and object classification. It does not provide a complete characterization of each object, such as a phase image, an amplitude image, or an accurate segmentation mask—information that is nevertheless valuable for detailed morphological analysis.

[0007] Furthermore, the method described in the aforementioned document is only functional for a fixed and known distance between the object and the camera. However, automatic focus finding (autofocus) is also a crucial step because it determines the distance between the object and the camera (denoted z), which is necessary for holographic reconstruction and also for tracking the object's movement. Moreover, in an online holographic configuration where the reference wave and the diffracted wave propagate along the same axis, backpropagation of the hologram in the plane of focus provides a focused image of the object, to which is added a second, defocused image of the conjugate object. However, this second image, called the twin image, complicates automatic focus finding with current algorithms.

[0008] Furthermore, to accurately delineate an object within the associated image and deduce information such as orientation or size, segmentation using a segmentation mask is necessary. However, the quality of this information depends largely on the performance of the segmentation algorithms used, and currently, no physics-based algorithm exists that can accurately segment holographically imaged objects.

[0009] Finally, another goal of holograms is to obtain automated classification of detected objects. For example, for planktonic cells, the classification can provide the name of the taxon or species. In the literature, particle classification in holograms is often performed using a trained convolutional neural network (CNN) model. The input images for this model are, in most cases, regions of interest (ROIs) extracted from the focused reconstructed images. However, classification with a CNN is an image classification task. If several cells are present in the same input ROI, the CNN will not be able to classify all the objects, especially in the case of dense samples. Furthermore, the classification step necessarily requires prior localization of the objects, which adds computation time.

[0010] Thus, to this day, we are still seeking to make the processing steps of the different tasks of a holographic characterization method faster and to make the computational resources used low in order to integrate them into compact optical holographic measurement devices, in particular the resources would be of the nano-computer type.

[0011] Furthermore, there is currently no digital processing method that allows for the simultaneous and real-time performance of all tasks related to holographic reconstruction, segmentation, localization, and object classification.

[0012] The invention therefore aims to overcome the aforementioned drawbacks, in particular by proposing a holographic method for characterizing (phase image, amplitude image and segmentation mask as well as classification and position) several microscopic objects in a sample, the method being automated and delivering real-time characterization results.

[0013] According to the invention, the method for characterizing several microscopic objects in a sample to be studied by digital holography, which extends in space along perpendicular axes x, y, z, the sample being illuminated and observed by a camera positioned at a distance from the sample along the z-axis, comprises: - a step of capturing an image of the sample by the camera to provide a raw hologram of the sample; - a step of processing the raw hologram of the sample, implemented by electronic processing means, characterized in that: - the step of processing the raw hologram of the sample, implemented by electronic processing means preferably of the nano-computer type, lists each of the objects of the raw hologram of the sample.in particular by automatically drawing a bounding box (also called ROI for "Region of Interest") around each object of the raw hologram of the sample, and, independently of knowledge of the distance z separating the sample from the camera and the distance separating each object from the camera (i.e., independently of focusing on each of the objects), implements a prediction model trained by machine learning, capable of deducing in real time, automatically and simultaneously, a characterization of each of the listed objects of the raw hologram of the sample (corresponding to the image taken of the sample), the characterization including for each object of the raw hologram of the sample: - a reconstructed phase image and a reconstructed amplitude image of each object,- a segmentation mask for each object (the segmentation mask is a binary mask used to remove the twin image; the segmentation mask also allows for the precise determination of morphological characteristics and the orientation of objects), (the reconstructed phase image, the reconstructed amplitude image, and the segmentation mask are provided as outputs of the prediction model), - a classification of each object, including a name assigned to the object, and - the object's position in the sample.

[0014] Depending on one characteristic, the method is implemented on a static surface sample, a static fluidic sample such as a static volume of liquid, or a sample in the form of a flow in a fluidic channel. Depending on the sample size and the camera's field of view, a single image of the sample is captured if the sample size corresponds to the camera's field of view, or several successive images of adjacent areas of the sample are captured if the sample is larger than the camera's field of view.

[0015] In the following description, "classifying" an object, or "classification," refers to assigning it a specific category, with each category corresponding to a name. The database contains numerous object categories, each identified by a name. Objects that cannot be classified will be grouped into an "unknown" category and subsequently characterized by experts and through manual data entry before being added to the database. Unknown objects can be automatically classified based on morphological similarities or other characteristics extracted from amplitude and phase images, or velocity profiles. It may be useful to deduce velocity profiles for moving objects.The velocity and trajectory profile of an object in a three-dimensional volume can, for example, be estimated from the bounding boxes of the object deduced from the video images taken by the camera and from tracking methods known in themselves such as the Kalman filter.

[0016] Thus, the holographic characterization method of the invention for multiple microscopic objects allows—regardless of the camera-sample distance when the image is captured—for significant processing time savings and the real-time provision of various characterization details for the objects present in the sample under study. This is achieved by automatically and simultaneously producing not only a reconstruction of the phase and amplitude images of each object in the sample, but also several other pieces of information, including the segmentation mask, a bounding box, the classification of each object, and the location of the identified (classified) object within the sample. "Real-time" refers to a duration of at most a few seconds (at least currently, based on existing electronic processing capabilities integrated into the imaging device).Therefore, the method of the invention does not sequentially implement different methods or models for each feature to be deduced, which would result in processing times of several minutes. Instead, it has the advantage of implementing a predictive model capable of deducing, in a few seconds, the holographic reconstruction of an object's amplitude and phase, as well as other features including a bounding box, the segmentation mask, and the classification. It should also be noted that this deduction of the reconstructed hologram's amplitude and phase (in just a few seconds) is achieved even when the image capture yielded a raw hologram and the focus distance of the objects was unknown. It is worth recalling that the focus distance is essential for reconstructing an amplitude image and a phase image.The operator does not need to know or determine the camera-object distance to focus each object and identify it through focused image processing. Furthermore, the prediction for identifying each object in the sample is performed using a single raw hologram of the sample observed by the camera, rather than multiple raw holograms recorded by adjusting the camera distance (by moving the camera) to focus on each object. The prediction model can predict, for a single raw hologram without knowing the focus distance, an amplitude image, a phase image, and a mask, as well as the object's classification and position. The method of the invention is therefore significantly different from the aforementioned document by Romane Scherrer et al., which fails to generate a phase image, an amplitude image, and a segmentation mask, even without knowing the focus distance.

[0017] According to one example of application of the method, the characterization method is applied to the recognition of plankton species by placing a sample containing plankton, imaging the sample and from a single image taken of the sample (or part of the sample) corresponding to a single raw hologram of the sample, to be able to automatically and in real time deduce from this image the different species recognized of plankton in the raw hologram of the sample, and this independently of the knowledge or input of the distance separating the camera from the sample.According to one characteristic, the prediction method is implemented using a database of object classifications (e.g., plankton species) whose construction required prior human identification of the classifications (plankton species) with manual data entry, and the generation of numerous simulated holograms, the proportion of real data to simulated data being at most 15%, preferably at most 10%. In the following description, "raw hologram" means a hologram obtained directly from an image captured by a camera, and "simulated hologram" means a virtual hologram generated by a computer from the reconstructed amplitude and phase images of raw holograms.Designing a database of simulated holograms significantly reduces the time required to build a database and minimizes the risk of human error. Otherwise, the operator would have to examine and repeat the shooting, focusing, and data entry operations on tens of thousands of objects. Advantageously, from a single raw hologram, multiple simulated holograms can be generated. The design of the simulated hologram database further reduces database construction time and lowers the prediction error rate of the classification method of the invention. This method identifies microscopic objects and provides, for each object, a segmentation mask and amplitude and phase images.

[0018] According to one characteristic, the prediction model uses a database that has been built beforehand (i.e., before the processing of the raw hologram of the sample being studied and whose objects are to be identified), from simulated holograms of samples that have been generated by a computer from raw holograms of a multiplicity of samples and raw holograms of focused objects of these raw holograms of samples, each simulated hologram of a sample comprising several raw holograms of known objects that have been virtually (in particular randomly) positioned and oriented in multiple positions and orientations in x, y, z (positions and orientation different from the position and orientation that each object had in the initial raw hologram of the sample; orientation is understood by a rotation in particular in the x, y plane).According to one characteristic, the prediction model database was built by a computer that is separate from the electronic processing means (of the nano-computer type) of the raw hologram processing step of the sample to be studied.

[0019] According to one characteristic, the prediction model uses a database designed beforehand by generating simulated holograms of samples. The prior construction of the database of simulated holograms of samples involves the following steps for a sample: - Step 1: acquisition of a multiplicity of raw holograms of the sample from the camera shot at different z positions of the sample, the sample comprising several objects; - Step 2: processing of the raw holograms of the samples consisting of extracting ROIs, each ROI containing a single object, and characterizing each of the objects in the ROIs by determining the focus distance z of the object, extracting the segmentation mask of the object, its amplitude image, its phase image and assigning it a classification by manual entry;- Step 3: Construction of simulated holograms of samples for a multiplicity of samples to constitute input data for the prediction model; - Step 4: Training of the prediction model by machine learning.

[0020] Specifically regarding step 3, it comprises the following sub-steps: a) randomly selecting k objects from the ROIs; b) randomly positioning and orienting each of the k randomly selected objects in a plane denoted Z k in a volume of size NxN'xZmax, in particular N and N' correspond respectively to the number of pixels in x and y of the camera and Zmax is the maximum object / camera distance or the height of a simulated fluid column (where objects would be located); c) to randomly select k' positions (x,y) in each Z plane k for each object k ;d) to position and orient virtually each of the k objects in the k' positions (x,y, Z k) ;e) to simulate the transmission function for each of the k objects and its k' positions in x, y from the amplitude and phase images ;f) to simulate the transmission function in three dimensions in the position (x,y, Z k ) from the transmission function of step e) and the position in Z k to deduce a simulated hologram of a sample (simulated hologram of a sample comprising several objects in simulated positions and heights).

[0021] This simulation positions each object in different orientations of the same Z plane. k and by repeating the position changes again in other Z planes kPlans allow for the rapid and automated expansion of the training dataset without human intervention, saving time and improving data reliability. The multiplication of tasks that an individual would otherwise have to perform to obtain a database of tens of thousands of data points would highly lead to errors. Furthermore, it should be noted that the selection of z-positions of the Z-plans k between [0, Zmax] is preferably random.

[0022] In step 4 of machine learning (training the prediction model), it is specified that, in parallel with the construction of each simulated hologram, the following is deduced: - a single image called the amplitude projection, which groups the projection of all the amplitude images of the Z planes k - a single image called the phase projection, which combines the projection of all the phase images of the Z planes kand—a so-called mask projection image, which groups the projection of all the masks of the Z planes k The amplitude, phase, and mask projection images associated with each simulated hologram are saved. These simulated amplitude, phase, and mask projection images, stored in the database, are used as reference images for supervised model learning. They serve as the reference targets for optimizing model parameters for predicting amplitude, phase, and mask distributions.

[0023] Preferably, the prior construction of the database by simulated holograms includes an additional step 5: adding raw holograms from a new sample with objects, having previously selected z positions for which there had been no image acquisition in step 1, then making predictions of characterization of objects from the sample from the prediction model and verification on a visualization interface by a human operator (expert in the field of identification of said objects) of the predicted classification with that known by the human operator, and if the operator considers that the prediction is exact (correct), integration of the new raw holograms from the sample into the database with reiteration from step 2.When predicting the characterization of the objects in the sample, the prediction model delivers, by majority vote, the most probable characterization prediction, which is the one verified by the human operator. Preferably, the model also delivers minority votes for which the human operator can perform checks to validate that these minority votes correspond to bad predictions (in order to confirm the accuracy of the prediction model by your majority).

[0024] According to one characteristic, the prediction model is also capable of delivering (via the prediction model) the focus distance z fof each object in the sample studied, that is, the distance separating the object from the camera for which the object's hologram is focused and therefore sharp. This estimation or prediction of the focus distance is made possible through supervised learning, using a database of simulated hologram samples, for which the associated focus distances constitute target or reference distances used during training.

[0025] According to one characteristic, the raw holograms of a sample to be studied are acquired from images taken by the camera by moving the support in x and / or y for a given z when the camera's field of view is smaller than the sample to be studied.

[0026] According to one feature, the aforementioned method of the invention is implemented by a holographic imaging device comprising the camera and the electronic processing means which consist of a nano-computer (which is therefore integrated (embedded) in the imaging device, making the device compact while allowing real-time identification of all objects in the sample studied).

[0027] Depending on one characteristic, the method is implemented on a static surface sample or a static fluidic sample such as a static volume of liquid, or even on a sample in the form of a flow in a fluidic channel.

[0028] According to one characteristic, the method is applied to the recognition of plankton species from a sample containing plankton, the camera taking a single image of the sample corresponding to a single raw hologram of the sample, regardless of knowledge or input of the distance separating the camera from the sample, the method automatically and in real time deducing from the image the different recognized plankton species in the raw hologram of the sample.

[0029] According to one characteristic, the method can be used to deduce in particular the following information from objects: - velocity profile and 3D trajectories of objects that move naturally or forced in a microfluidic channel, - synthesis of the number of objects by classification in the sample, - extraction of amplitude and phase images of objects in the form of thumbnails grouping the following characteristics for each thumbnail: object classification, segmentation mask, reconstructed phase image, reconstructed amplitude image, - extraction of the morphological characteristics of objects, - separation of known objects identified by said method, from unknown objects also recorded in the sample by said method (such as new species, bubbles, dust, etc.), - pre-classification of unknown objects, in particular from extracted morphological data and / or trajectories.

[0030] The invention also relates to one or more computer programs comprising code instructions for executing the steps of the aforementioned method of digital holography characterization of the invention, when the code instructions are executed by a computer.In particular, the invention relates on the one hand to a first computer program comprising code instructions which, when executed by a first computer, in particular a nano-computer, cause the first computer to execute the steps of the aforementioned digital holography characterization method of the invention to characterize, in real time and in an automated manner from the prediction model, objects of a sample to be studied, and on the other hand, to a second computer program comprising code instructions which, when executed by a second computer distinct from the first computer, cause the second computer to build the database of the prediction model.

[0031] Finally, the invention relates to a holographic imaging device configured to characterize objects of a sample to be studied by digital holography, the imaging device comprising a light source and preferably a diaphragm coupled to the light source, a camera comprising an image sensor, the sample to be studied being placed between the light source and the camera, and electronic processing means, in particular a nano-computer, on which is installed a computer program of the aforementioned invention, the computer program comprising code instructions which, when executed by the computer, cause the computer to execute the steps of the aforementioned digital holography characterization method of the invention to characterize, in real time and in an automated manner from the prediction model, the objects of the sample to be studied.

[0032] According to one characteristic, the holographic imaging device comprises a single camera that is capable of being translated along the z-axis, and a support that is capable of being moved in the x and y directions.

[0033] The present invention is now described by means of purely illustrative and in no way limiting examples of the scope of the invention, and from the accompanying illustrations, in which: - represents a schematic view of the holographic imaging device according to the invention implementing the method of the invention; - illustrates an image captured by the camera of the imaging device, corresponding to a raw hologram of a part of the sample (the sample being larger than the field of view of the camera); - illustrates the real-time display of a reconstructed full-amplitude image of the sample where all objects appear focused with their bounding box, their classification and the focus distance.- [a] illustrates the display of an example of a characterization thumbnail associated with one of the objects of the raw hologram of the object automatically generated in real time, all the objects of the raw hologram being intended, thanks to the predictive model of the method of the invention, to be automatically and in real time, identified by their name (classification) while being associated with their characterization data (reconstructed phase image, reconstructed amplitude image, segmentation mask, position of the object)- [] illustrates for the prior construction of a database, a raw hologram of the sample, its segmentation mask, its amplitude image and its phase image.- [] illustrates a processing step of a raw hologram in the construction of the database, including the extraction of ROI and the characterization of the object of the ROI.- [] represents a schematic view of a step in generating a simulated hologram by positioning objects from a raw hologram acquired by the camera in virtual planes. - [] is a schematic view of an example of random positioning and random orientation of objects in a given volume.

[0034] The method for characterizing objects by digital holography according to the invention is implemented by a holographic imaging device 1, schematically illustrated in Figure 1, which incorporates electronic processing means of the nanocomputer type capable of executing an appropriate prediction model. The method aims to perform simultaneously and in real time different characterization steps of millimeter-sized or microscopic objects, preferably objects with a size between 5 µm and 1 mm, by deducing from the prediction model both the holographic reconstruction, the segmentation, the classification of each object in a sample 10 studied, and, if necessary, the localization of each object within the sample 10.Advantageously, the method of the invention also makes it possible to deduce the focus distance of each object, that is to say the distance at which the camera must be located relative to the object to be identified in the sample in order for the raw hologram of said object to be clear.

[0035] The characterization method of the invention is a predictive method designed to reduce object identification time by automatically performing real-time object identification. Furthermore, as will be seen later, the characterization method includes a preliminary step of constructing at least one database. This step saves time in developing said database through the simulation of holograms, which will be used to train the predictive model using machine learning. Preferably, the predictive model will be trained using only simulated hologram data; the raw holograms will be used to generate simulated holograms and will contribute to verifying predictions during training.The prediction model of the method of the invention uses raw holograms and simulated holograms as data, in particular the simulated holograms representing more than 80% of the database, preferably at least 90%. Furthermore, the prediction method has the advantage of not using a Fourier transform (the Fourier transform slowing down processing calculations).

[0036] The method is implemented by the imaging device 1, schematically illustrated on the diagram, which integrates the electronic processing means 1' of the nano-computer type. The nano-computer 1' therefore implements a computer program that includes code instructions (algorithms) capable of characterizing (identifying) the objects in the sample in an automated and real-time manner.

[0037] A sample 10, comprising several objects 10a, 10b, 10c, etc., to be identified, is positioned on a support 2 for observation by the imaging device 1. This device generates a raw hologram 11 of the sample and automatically deduces, in real time via the method of the invention, the characteristics of each object, relating to the holographic reconstruction of each object in the sample, its segmentation, classification, and location within the sample, even though no focusing operation is required on each individual object. A graphical interface is advantageously coupled to the imaging device 1 and allows the characterization of each object 10a, 10b, 10c, etc., to be displayed on a screen.in the form of a vignette 12 () associated with each object and grouping for the object, a reconstructed hologram of the object, a reconstructed phase image, a reconstructed amplitude image, a segmentation mask and a classification (“NAME” of the object, here A, B, C or D).

[0038] The image shown is the observed and captured image of sample 10 by the camera (the captured image corresponds to the camera's field of view, here 10 mm as an example). 2This captured image constitutes a raw hologram 11 of the sample (in particular, a portion of the sample) which the operator visualizes on the graphical interface. From the image capture, the method of the invention, implemented by the imaging device 1, thus makes it possible to visualize on a display screen a raw hologram 11 (of a portion) of the sample 10 with several objects to be identified. In an automated manner, the nanocomputer 1' will select (enclose each of the objects in the raw hologram of the sample) and automatically identify each of these objects in real time. The operator can, for example, display on their display screen an amplitude image of the sample ( ) where all the objects appear focused with their bounding box, their classification, and the focus distance.The operator can also display, on their viewing screen, for each object linked to its location in the sample, an associated thumbnail 12 characterizing that object, including the reconstructed hologram of the object, the reconstructed amplitude image of the object, the reconstructed phase image, its segmentation mask, and its classification, such as the object's name (NAME). The reconstructed hologram of the object corresponds to a hologram from the database without the operator needing to know the distance of the sample from the camera or to use focusing to obtain a clear image of the object to be identified, and this applies to all objects in the sample.With a single image taken of several objects within the camera's field of view, regardless of the distance from the sample to the camera and regardless of the distance of each object to be identified from the camera, the method of the invention automatically and in real time generates a characterization vignette associated with each object and a location of the identified object within the sample. The characterization vignette 12 includes the reconstructed hologram of the object, the mask, the amplitude, the phase, and the object's name.

[0039] The imaging device 1 illustrated on lasert both implements the method of characterization by prediction of objects in a sample in real time, and implements a prior step of building a database to implement the prediction model of said characterization method of the invention.

[0040] The imaging device 1 comprises a support 2 for arranging the sample 10 in an (x, y) plane, a light source 3, and preferably a diaphragm 4 arranged on one side of the sample 10, a camera 5 positioned on the other side of the sample 10 opposite the light source 3, and the nanocomputer 1'. The support 2 is capable of movement in the x and y directions; indeed, the sample 10 (for example, a slide) will present a larger surface area than the field of view of the camera 5, and moving the support will allow images to be obtained of the entire sample or of specific areas. Furthermore, the camera 5 is capable of movement in a direction perpendicular to the plane of the sample, that is, in a z-direction perpendicular to the x and y directions. A plane P0 is taken as a reference and is perpendicular to the z-axis.The mobility of the camera 5 serves in the preliminary step of building the database to generate different raw holograms at different focus distances of different objects, as well as to update the database for objects not recognized by the method of the invention and which will therefore have to be identified and added to the database.

[0041] The sample 10 comprises several objects 10a, 10b, 10c, etc. to be identified and characterized, such as cells, dust, pollen, or microorganisms. These particles can be placed on a microscope slide, a cuvette, or a microfluidic channel.

[0042] The support 2 can be translated along the x and y axes, for example by two respective slide systems 20 and 21. The movement can be carried out manually by the user or automated by linear actuators or motors.

[0043] Holography is preferably inline, meaning that the propagation direction of the light source 3 is perpendicular to the plane P0, along a z-axis perpendicular to the x and y axes. The light source 3 is, for example, a green light-emitting diode (LED) with a central wavelength of λ = 520 nm and a spectral band Δλ ≈ 20 nm. This LED 3 is spatially filtered by the diaphragm 4, which has a diameter of, for example, 100 µm, to increase the spatial coherence of the light source. Objects 10a and following are illuminated by a wave 3', called the illumination wave, emanating from the LED 3 and filtered by the diaphragm 4. The distance between the diaphragm 4 and the objects is a few centimeters, generally between 5 and 10 cm, to ensure that the illumination wave is plane-bound in the plane of the objects. Each object such as 10a has its own transmission function denoted

[0044]

[0045] which modulates the illumination wave in amplitude and phase. The illumination wave is diffracted by the sample, producing a diffracted wave 3'' which propagates along the optical axis z.

[0046] The camera 5 includes an image sensor 50 located in a plane P1 perpendicular to the propagation axis z. The sensor 50 is composed of a CMOS or CCD pixel array. If the camera is equipped with a Bayer filter, choosing a green or infrared LED 3 optimizes the resolution of the acquired images. The camera 5, via its sensor 50, captures an image like the one shown, and the nanocomputer 1' is capable of displaying on the viewing screen the raw holographic image 11 (raw hologram of the image of the camera's field of view) resulting from the interference of the reference wave with the waves diffracted by objects 10a and others.

[0047] The camera 5 is capable of being translated along the z-axis. In particular, the camera is mounted on a removable support 6 that can translate along the z-axis by means, for example, of a system comprising a threaded rod 60 and a nut 61. The threaded rod is preferably actuated by a stepper motor 62 via a rigid or flexible coupling. The minimum displacement of the support 6 along the z-axis is determined as a function of the number of motor steps and the pitch of the rod 60. The minimum displacement is, for example, on the order of 4 µm. This displacement can be reduced by means of a suitable electronic motor control system 63. The displacement of the camera support 6 is limited in the immediate vicinity of the sample support 10 to prevent it from colliding with it; this limitation is, for example, controlled by a limit switch 7.

[0048] The nanocomputer 1' controls the camera 5, the light source 3, and the stepper motor 62. The nanocomputer 1' is capable of implementing the object characterization (and therefore identification) method according to the invention. The nanocomputer 1' also incorporates the graphical interface that allows the user to adjust the control parameters of the elements 3, 5, and 62 of the imaging device 1, to view the images on a display screen showing the raw holograms of the images captured by the camera and the characterization results of each object in a sample studied by the camera, as well as to manage the saving of images or videos and results.

[0049] The raw holograms studied correspond to a portion of the sample size 10, for example a size of 3280 × 2464 pixels, which corresponds to a field of view of 10 mm 2The camera used, for example, is equipped with a sensor whose pixel size is equal to 1.12 μm.

[0050] The method for characterizing objects 10a and following of the sample 10 according to the invention comprises the following steps: - placing the sample 10 here on the support 2 because the observed sample is a flat substrate such as a substrate slice, and this is irrelevant to the distance of the camera 5, i.e. without knowing the z-position of the camera; in the plane of the support, instead of a slice of the substrate, a static fluidic sample such as a static volume of liquid or a flow in a fluidic channel circulating in a direction of the plane of the support could be observed, - illuminating the sample 10 by the light source 3, - capturing and processing an image of the sample or a part of the sample by the camera 5, such as the image illustrated in Figure 1, which constitutes a raw hologram 11 of the sample (or a part of the sample), - automated outlining by the nano-computer 1'.of each object of the raw hologram 11 of the sample according to a bounding box or ROI,- real-time production of the characterization and position in the sample of each of the objects 10a and following of the raw hologram of the sample from an appropriate prediction model and at least one database, integrated into the nanocomputer 1', said at least one database having been previously constructed and comprising simulated holograms (in particular tens of thousands of simulated holograms) of samples containing objects already known,- display on a viewing screen for the user, optionally of the raw hologram 11 of the sample, optionally of a reconstructed amplitude image where all objects appear focused with their classification and their focus distance, and of at least one characterization vignette 12 of one of the objects of the sample, the vignette comprising the reconstructed hologram of the object,The reconstructed amplitude image of the object, the reconstructed phase image of the object, the segmentation mask, and the classification. The graphical interface will be designed according to the user's needs to visualize information such as multiple side-by-side thumbnails of different identified objects, map objects of the same classification, provide statistics on objects, etc.

[0051] For example, the sample 10 to be studied comprises several objects belonging to one or more plankton species that we wish to identify automatically. With a single image taken from the sample, the method of the invention makes it possible to identify each plankton by automatically assigning it, in real time, a vignette including the plankton species name, the reconstructed hologram of the plankton, the reconstructed amplitude and phase images of the plankton, and the segmentation mask. The x, y, z position of each identified plankton in the sample is also provided by the method of the invention.

[0052] The prediction model that produces the object characterization was designed in a preliminary phase, using machine learning from a database specifically constructed with simulated holograms. The preliminary phase of constructing this database (at least one database) comprises the following steps: - Step 1: Acquisition of multiple raw holograms from several samples (corresponding to multiple images taken from each sample), with each sample containing several objects distributed across it. A plurality of raw holograms are acquired for a defined area in x, y coordinates (each hologram corresponding to only 10 mm). 2of the sample), while varying the distance z from a sample to the camera (a given z corresponds to a raw hologram). By moving the sample within the plane of the support to capture another area in x, y, several other raw holograms of the sample are also acquired. The raw holograms of samples acquired in this step 1 are two-dimensional images corresponding to the area captured by the camera sensor 5. A holographic dataset is stored (in a database) and thus corresponds to several raw holograms of a sample, each acquired for a different z, and this for several samples.Each holographic dataset includes, as illustrated in the figure, a) the raw hologram of the sample, b) the corresponding segmentation mask, c) the corresponding amplitude image, d) the corresponding phase image; - Step 2 (): processing of the raw sample holograms consisting of manually extracting ROIs, each ROI containing a unique object, and characterizing each of the objects in the ROIs by determining the focus distance z of the object, extracting the segmentation mask of the object, its amplitude image, its phase image and assigning it (manually) a classification, for example A, B, C, D, etc.The ROIs, along with the characterization (phase image, amplitude image, and segmentation mask) of each object in the raw holograms, are recorded in a database; - Step 3: production of simulated holograms to constitute input data for the prediction model and to supplement the raw holograms in order to balance the data for machine learning of the prediction model; - Step 4: training of the prediction model using machine learning to obtain a high-performing prediction model. The prediction model and the aforementioned database were designed using appropriate code instructions (algorithms) and based on the acquisition of raw holograms and the simulation of so-called simulated holograms from the raw holograms.- Preferably, step 5: adding raw holograms from a new sample, having previously selected z-positions for which there was no image acquisition in step 1 (in order to rebalance the z-distribution), then making predictions for characterizing objects from the sample based on the prediction model and verifying by the operator on the graphical interface the predicted classification with that known to the expert, and if the prediction is true, integrating the new raw holograms into the database with reiteration of step 2.

[0053] Each of steps 1 through 5 is now described in more detail. Preferably, the prediction model is implemented using only simulated holograms.

[0054] Step 1 involves an operator positioning a sample on support 2, and the imaging device 1 acquiring raw holograms. The sample contains various objects, such as plankton, whose concentration varies from one sample to another. For example, a sample may contain dozens or even hundreds of plankton. For a single sample, a series of raw holograms is acquired by modifying (automated control) the distance between the sample containing the plankton and the camera by moving the camera along the z-axis, for example, in 20 µm increments. This raw hologram acquisition step is performed on hundreds of different samples (each time a different sample is placed on support 2). Furthermore, the operator knows that the set of samples contains different plankton species, for example, four different species A, B, C, and D.Each raw hologram of a sample can contain dozens or even hundreds of plankton, depending on the sample concentration, and may include one or more plankton species. In total, several hundred raw holograms are acquired; Figure a) shows an example of a raw hologram acquired from a sample. The acquired raw holograms are processed automatically by electronic processing means to associate each with its segmentation mask (b), amplitude image (c), and phase image (d). In particular, the raw sample holograms are reconstructed using classical reconstruction methods from the image captured by the camera, the inferred segmentation mask, and the focus distance z. f known corresponding to the associated raw hologram.

[0055] Step 2 () consists of using the interface to characterize the objects in each of the raw holograms, by extracting ROIs from these raw holograms, each ROI containing a single object (a single plankton) (two ROIs are identified on the image, for example), and by characterizing each object within an ROI. The acquired raw holograms of objects (plankton) from the sample correspond to holograms for which the operator has sharpened (focused in the z-axis) the object. In particular, sharpening each object in an ROI is done manually by the operator, for example by scrolling the depth of field using a computer mouse until the object is sharp on the viewing screen. For each ROI for which an object (such as plankton) is focused, the operator draws a bounding box around the object; Extracting a ROI corresponds to a bounding box.In particular, the bounding box in which an object is inscribed determines the following parameters: the x and y position of the box relative to a zero reference, the width w of the box, and the length h of the box. The operator assigns the object a classification such as the species name (which they know from among the names A to D). Each classified object in a bounding box has a raw hologram whose z-position is known, and the nanocomputer is capable of automatically processing this raw hologram of the object to extract the segmentation mask and the amplitude and phase images of said classified object. Since the z-position of the object is known, the phase and amplitude extractions are performed in the usual way, for example, using the so-called "dual-plane phase recovery" method. The operator repeats the operation of delimiting an object by a classification bounding box for each of the other objects in the raw hologram of the sample.Each object in a bounding box is automatically associated with characterization data: the object's raw hologram, the segmentation mask, amplitude and phase images, and the name (A, B, C, or D). This illustrates four examples of ROIs (Return of Interest) with their characterization: amplitude and phase images extracted from raw holograms and recorded for four different planktonic objects, each corresponding to a different plankton species (A through D). The characterization data for each classified object is recorded in a database, along with the bounding box coordinates of each classified object within the sample. The operator thus characterizes several objects found in each raw hologram of the sample and repeats the characterization of objects contained in other samples.For example, the number of samples studied is 800, and a sample can have a different concentration and include, in particular, between 10 and 500 planktonic objects. Therefore, at least 800 raw holograms of samples are acquired (depending on the sample size). Furthermore, four known plankton species, A to D, are considered, and for each of the four species, 1000 different planktonic individuals (1000 ROIs) are extracted. For each object, a characterization vignette is provided, including the raw hologram of the object, its segmentation mask, its amplitude image, its phase image, and the plankton species. Ultimately, the database comprises 4000 planktonic objects with their characterization vignettes (segmentation mask, amplitude image, phase image, classification).The operator therefore performed 4000 manual acquisition operations (focusing on an object, outlining and entering the classification) not counting the setup of 800 samples.

[0056] Step 3 involves populating the databases with more data (a greater number of images) so that the invention's method for characterizing and recognizing objects can subsequently be performed automatically from a single image. Advantageously, the prediction model was trained using machine learning on a database of simulated holograms. Indeed, for machine learning on plankton data, tens of thousands of holograms are needed for the characterization model to be sufficiently trained. However, steps 1 and 2 above, which already require a significant amount of work from an operator, only provide a few thousand raw hologram samples within a reasonable timeframe, which remains insufficient for a plankton database.The database is necessarily unbalanced (some species are more represented than others, and the distribution of object / camera distances is not uniform) and lacks sufficient images to train a model. The inventive feature of step 3 in constructing the database for the prediction model of the invention is therefore to simulate holograms to constitute additional input data for the prediction model. These so-called simulated sample holograms are created from the raw holograms of the ROI objects already recorded in the database. The simulated sample holograms will become the input images for the model in order to obtain a balanced dataset.

[0057] Thus, starting with a few thousand raw holograms of objects, this simulation step makes it possible to produce tens of thousands of simulated holograms of samples containing mixed objects. The simulated holograms are obtained by automatically changing the position and quantity of objects within a given volume to provide a simulated hologram of a sample in which the objects are in different positions and at different heights. Learning from simulated holograms results in significant time savings in database construction. In particular, it saves an operator from performing tens of thousands of operations that would involve taking holographic measurements by successively changing the distance z between the camera and the sample and considering tens of thousands of different samples.As an example, the operator will create 800 raw holograms of objects by classifying them, and 40,000 simulated holograms of samples incorporating these objects in various positions and heights will be automatically generated, instead of manually performing 40,000 manipulations.

[0058] More specifically, the production of simulated holograms involves the following substeps implemented by a computer and carried out by a computer program containing code instructions capable, from raw holograms of objects: a) of randomly selecting k objects from the ROI (for example, k planktonic cells from 4000), b) of randomly positioning each of the k randomly selected objects in a plane denoted Z kin a volume of size NxN'xZmax(), where N and N' correspond respectively to the number of pixels in x, y of the camera and Zmax is, for example, the maximum object / camera distance (that the learning model will be able to find after training) or, for example, the height of a simulated water column; c) to randomly select k' positions (x,y) in each plane Z k for each object k ;d) to virtually position each of the k objects in the k' positions (x,y, Z k ) – that is to say, the object is positioned randomly and successively in a different position in the Z plane k e) to simulate the transmission function for each of the k objects and its k' positions in x, y from the amplitude and phase images; f) to simulate the transmission function in three dimensions in the position (x, y, Zk ) from the transmission function of step e) and the position in Z k to deduce a simulated hologram for a sample.

[0059] In parallel with step d) above, which involves producing and saving a simulated hologram and saving the ROIs and classifications, three projection images are produced and recorded, associated with said simulated hologram, in a single plane. These are: - an amplitude projection image, which corresponds to the projection of all the amplitude images of the Z planes k - a so-called phase projection image, which corresponds to the projection of all the phase images of the Z planes k and an image called a mask projection, which corresponds to the projection of all the masks of the Z planes kTherefore, for each simulated hologram, four images are saved: the simulated hologram, the amplitude projection image, the phase projection image, and the mask projection image; these saved images will serve as reference images or targets in the learning (i.e., training) phase of the prediction model to deduce (predict) the reconstructed amplitude image, the reconstructed phase image, and the segmentation mask.

[0060] Therefore, starting from a raw hologram of an object initially arranged in a defined position (x,y,z) and which has been classified, a multitude of different simulated holograms are automatically generated from a sample comprising a multitude of objects (raw object holograms) in different positions along (x,y,z). Each simulated sample hologram has its own segmentation mask projection image, phase projection image, and amplitude projection image (four images are thus saved per simulated hologram: the simulated hologram, the mask projection image, the amplitude projection image, and the mask projection image). The simulated sample holograms are integrated into the database and thus become input images for the prediction model.In the example considered, from 1000 raw holograms of objects multiplied by 4 different species A to D, i.e. 4000 objects, substeps a) to f) (which implement adapted code instructions) make it possible in particular to generate 40,000 simulated holograms of samples, each of the 40,000 simulated holograms being different from the others.

[0061] During training, prediction results will be compared with target values ​​and depending on the score on the calculated error, the parameters of the prediction model can be updated.

[0062] Step 4 of the machine learning process involves training the prediction model based solely on simulated holograms (40,000 holograms). Each simulated hologram sample therefore contains different objects in different random positions, providing a balanced training set for the prediction model, particularly regarding the distribution of object / camera distances. Specifically, training on a computer equipped with a graphics processing unit (GPU), such as the Tesla V100 GPU, takes approximately ten hours for training from 30,000 to 40,000 input data points over 600 epochs, with a batch size of 32 and a learning rate of 0.1.

[0063] Preferably, the learning stage verifies that the trained model is performing well by analyzing its predictions, that is, by analyzing results delivered by the characterization method (delivery of vignettes (with phase image, amplitude image and mask) and recognition for example of a plankton species) from the database comprising only simulated holograms of samples (and the projection images of amplitude, phase and mask), by comparison with results from simulated holograms that had not yet been introduced into said database of simulated holograms and / or by comparison with raw holograms whose objects have been identified by the human eye of an expert.

[0064] Preferably, an additional step 5 further improves the prediction model. This step 5 allows for continued training to implement new simulated holograms from these new raw sample holograms and to rebalance the dataset of raw object holograms. This step 5 includes the following substeps: - a new sample (for example, a static sample in the form of a slide) is positioned on support 2 of device 1; - the nanocomputer 1' selects object / camera positions to balance the distribution of z-positions in the database. Indeed, the nanocomputer 1' has information related to the distribution of the databases; however, the distribution of object z-positions is initially unbalanced and therefore does not follow a uniform distribution.The goal is to rebalance the distribution with respect to the z-positions by obtaining holograms for z-positions where no images were initially captured. Based on each automatic selection of a z-position, the camera moves along the z-axis and captures a raw hologram of the sample at each selected position. The trained prediction model then performs predictions for each new raw hologram of the sample, generating characterization data such as the bounding box, phase image, amplitude image, segmentation mask, classification, and the (x, y, z) position of each object.For each new sample, new raw holograms of objects (for example, about twenty) are generated at different z-positions or "heights" to rebalance the distribution and the associated predictions from the model. A majority vote is then performed (based on the classifications predicted for each object in the sample) to refine the classification predicted by the model for the detected objects. To do this, the classification provided by the prediction model is compared, for each object and for each raw hologram in the sample at different heights. The class predicted by the majority vote is retained as the "true" or "correct" class for the object. The prediction by majority vote will be verified by a human operator.Preferably, the model also delivers minority votes for which the human operator can perform checks to validate that these minority votes correspond to bad predictions in order to confirm the accuracy of the prediction model by your majority; - finally, the nano-computer updates the distribution of z positions in the database by integrating the new raw holograms of the objects.

[0065] Step 5 is preferably performed for each new sample.

[0066] Preferably, raw object holograms and simulated holograms of prediction model samples are stored in separate databases. New raw object holograms with a new classification must be verified by an expert before being integrated into the database. For this purpose, the raw object holograms, along with predictions of amplitude and phase images, segmentation masks (converted to polygons), and the classification prediction, are displayed on the visualization interface. Experts visually verify the predictions (particularly the mask and classification), and may make modifications, notably to the classification and / or by redrawing the bounding box polygon.

[0067] Furthermore, when the method fails to classify an object in a sample, either the classification is not in the database, or the classification is in the database but the prediction model failed. Step 5 is then systematically implemented first. If, ultimately, the classification is not listed in the database (for example, a new plankton species E distinct from the existing classifications A to D in the database), the database must be updated, and the same initial database construction steps 1 to 4 will be implemented to generate simulated holograms of samples. Preferably, these steps will be followed by step 5.

Claims

Method for characterizing by digital holography several microscopic objects (10a, 10b, 10c) in a sample to be studied which extends in space along mutually perpendicular axes, x, y, z, the sample being illuminated and observed by a camera (5) disposed at a distance from the sample along the z-axis, the method comprising: - a step of capturing an image of the sample by the camera to provide a raw hologram of the sample, - a step of processing the raw hologram of the sample, implemented by electronic processing means, characterized in that the step of processing the raw hologram of the sample, implemented by electronic processing means preferably of the nano-computer type, lists each of the objects of the raw hologram of the sample, in particular by automatically drawing a bounding box around each object, andIndependently of knowledge of the distance z separating the sample from the camera and the distance separating each object from the camera, implements a prediction model trained by machine learning, capable of deducing in real time, automatically and simultaneously, a characterization of each of the listed objects in the raw hologram of the sample, the characterization including for each object in the raw hologram of the sample: - a reconstructed phase image and a reconstructed amplitude image of each object, - a segmentation mask of each object, - a classification of each object, including a name assigned to the object, and - the position of the object in the sample. Method according to claim 1, characterized in that the prediction model uses a database having been previously constructed, from simulated holograms of samples which were generated by a computer from raw holograms of a multiplicity of samples and raw holograms of focused objects of these raw holograms of samples, each simulated hologram of sample comprising several raw holograms of known objects which have been virtually positioned and oriented in multiple positions and orientations in x, y, z. Method according to claim 1 or 2, characterized in that the prediction model uses a database designed beforehand by the generation of simulated holograms of samples, the prior construction of the database of simulated holograms of samples comprises the following steps for a sample:- Step 1: acquisition of a multiplicity of raw holograms of the sample from the image taken by the camera at different z positions of the sample, the sample comprising several objects;- Step 2: processing of the raw holograms of the samples consisting of extracting ROIs, each ROI containing a single object, and characterizing each of the objects of the ROIs by having determined the focus distance z of the object, by extracting the segmentation mask of the object, its amplitude image, its phase image and by assigning it by manual entry a classification;- Step 3: Construction of simulated holograms of samples for a multiplicity of samples to constitute input data for the prediction model; - Step 4: Training of the prediction model by machine learning. A method according to the preceding claim, characterized in that step 3 comprises the following substeps: a) randomly selecting k objects from the ROIs; b) randomly positioning and orienting each of the k randomly selected objects in a plane denoted Z k in a volume of size NxN'xZmax, in particular N and N' correspond respectively to the number of pixels in x and y of the camera and Zmax is the maximum object / camera distance or the height of a simulated fluid column; c) to randomly select k' positions (x,y) in each Z plane k for each object k ;d) to position and orient virtually each of the k objects in the k' positions (x,y, Z k) ;e) to simulate the transmission function for each of the k objects and its k' positions in x, y from the amplitude and phase images ;f) to simulate the transmission function in three dimensions in the position (x,y, Z k ) from the transmission function of step e) and the position in Z k to deduce a simulated hologram of a sample. Method according to claim 3 or 4, characterized in that the prior construction of the database by simulated holograms includes an additional step 5: adding raw holograms of a new sample with objects, having previously selected z positions for which there had been no image acquisition in step 1, then making predictions of characterization of objects of the sample from the prediction model and verification on a visualization interface by a human operator of the predicted classification with that known by the human operator, and if the operator considers that the prediction is correct, integration of the new raw holograms of the sample into the database with reiteration from step 2. Method according to the preceding claim, characterized in that during the prediction of characterization of the objects of the sample, the prediction model delivers by majority vote a most probable characterization prediction which is the one verified by the human operator, preferably the model also delivers minority votes for which the human operator can make checks in order to validate that these minority votes correspond to bad predictions. A method according to any one of the preceding claims, characterized in that the prediction model is capable of deducing the focus distance z f of each object in the sample studied. Method according to any one of the preceding claims, characterized in that the raw holograms of a sample to be studied are acquired from images taken by the camera by moving the support in x and / or y for a given z when the field of view of the camera is smaller than the sample to be studied. Method according to any one of the preceding claims, characterized in that it is implemented on a static surface sample or a static fluidic sample such as a static volume of liquid, or on a sample in the form of a flow in a fluidic channel. Method according to any one of the preceding claims, characterized in that it is applied to the recognition of plankton species from a sample containing plankton, the camera taking a single image of the sample corresponding to a single raw hologram of the sample, independently of knowledge or input of the distance separating the camera from the sample, the method automatically and in real time deducing from the image the different recognized plankton species in the raw hologram of the sample. Method according to any one of the preceding claims, characterized in that it is used to deduce the following information from objects: - velocity profile and 3D trajectories of objects moving naturally or forced in a microfluidic channel, - synthesis of the number of objects by classification in the sample, - extraction of amplitude and phase images of objects in the form of thumbnails (12) grouping the following characteristics for each thumbnail: classification of the object, segmentation mask, reconstructed phase image, reconstructed amplitude image, - extraction of the morphological characteristics of the objects, - separation of known objects identified by said method from unknown objects also recorded in the sample by said method, - pre-classification of unknown objects, in particular from extracted morphological data and / or trajectories. Computer program(s) comprising code instructions for executing the steps of the digital holography characterization method according to any one of claims 1 to 11, when the code instructions are executed by a computer. Holographic imaging device (1) configured to characterize by digital holography objects of a sample to be studied, the imaging device comprising a light source (3) and preferably a diaphragm (4) coupled to the light source, a camera (5) comprising an image sensor (50), the sample to be studied being placed between the light source and the camera, and electronic processing means (1'), in particular a nano-computer, on which is installed a computer program of claim 12. Holographic imaging device (1) according to the preceding claim comprises a single camera (5) which is capable of being translated along the z-axis, and a support (2) which is capable of being moved in the x and y directions.