Single fibre endoscope

Abstract

A method of operating an imaging apparatus, the method comprising: pre-training a machine learning model based on correlating proximal output image data associated with a proximal end of an imaging probe with distal input image data associated with a distal end of the imaging probe; determining proximal output image data associated with the proximal end of the imaging probe; using the pre-trained machine learning model to predict distal input image data associated with the distal end of the imaging probe based on the determined proximal output image data; and updating the machine learning model based on the predicted input distal image data and the determined proximal output image data to provide an updated machine learning model.

Description

SINGLE FIBRE ENDOSCOPEFIELD OF THE INVENTION

[0001] The invention relates to methods and apparatuses for imaging. In particular, but not exclusively, the invention relates to methods and apparatuses that use machine learning models to overcome challenges when imaging with imaging probes.BACKGROUND

[0002] Endoscopes made of a single multimode optical fibre are predicted to transform the field of clinical endoscopy because they are ultrathin (< 300 pm), high-resolution (< 1 pm, 10 times better than the conventional fibre-bundle based endoscope and can reveal cellular features), scanner-free and wide-field, and particularly suitable in minimally invasive in vivo diagnosis and imaging. Despite the intensified research interest in this topic, a singlemultimode -fibre endoscope that can be used in clinical procedures has yet to be delivered. The key unsolved challenge, faced by the global research community, is that the single-multimode-fibre imaging probes cannot be flexible (or even slightly disturbed), i.e., the flexibility challenge. This is because the image reconstruction via a multimode fibre relies on the precalibrated transmission properties of the fibre obtained when both ends of the fibre are accessible. During an endoscopic procedure, such fibre transmission properties change when the fibre is bent or moved (or even stationary), whilst it is impossible to access the fibre distal end for recalibration. It is an object of the disclosure to at least partly address one or more of the shortcomings in the prior art.SUMMARY OF THE INVENTION

[0003] According to an aspect of the invention, there is provided a method of operating an imaging apparatus, the method comprising: pre-training a machine learning model based on correlating proximal output image data associated with a proximal end of an imaging probe with distal input image data associated with a distal end of the imaging probe; determining proximal output image data associated with the proximal end of the imaging probe; using the pre-trained machine learning model to predict distal input image data associated with the distal end of the imaging probe based on the determined proximal output image data; and updating the machine learning model based on the predicted input distal image data and the determined proximal output image data to provide an updated machine learning model.

[0004] Recalibration of the machine learning model based on determined proximal output image data associated with the proximal end of the imaging probe and predicted input image data enables the imaging probe to be moved whilst still providing accurate predictions of input distal image data. This means that an imaging probe can be used to image at a location where recalibration is otherwise not possible due to lack of access to the distal end of the imaging probe.

[0005] Optionally, the method comprises iteratively adjusting the imaging probe and for each of one or more iterations: determining further proximal output image data from the proximal end of the imaging probe; using the updated machine learning model to predict further distal input image data associated with the distal end of the imaging probe based on the determined further proximal output image data; and updating the updated machine learning model based on the predicted further distal input image data and the determined further proximal output image data to provide a further updated machine learning model. Such iterative adjustment enables real-time imaging in situations where access to the distal end of the imaging probe is not possible. This addresses a problem that arises when the imaging probe must be inserted into a location and the correspondence between distal input image data and proximal output image data would otherwise break down. This means that the imaging probe can be moved between multiple imaging locations whilst maintaining a high imaging accuracy.

[0006] Optionally, the method comprises pre-training the machine learning model by coupling electromagnetic radiation into the imaging probe and detecting the coupled electromagnetic radiation at each of the proximal end and the distal end of the imaging probe. Optionally, the method comprises determining a transmission matrix associated with proximal output image data and corresponding distal input image data. Optionally, coupling electromagnetic radiation into the fibre comprises modulating the electromagnetic radiation to form a plurality of modulation patterns; and correlating each modulation pattern of the plurality of modulation patterns with a respective transmission matrix associated with proximal output image data and corresponding distal input image. By training the machine learning model on input-output pairs, the machine learning model can be used to predict distal input image data based on detected proximal output image data. Access to the distal end of the imaging probe is not required, yet accuracy in prediction through recalibration means that imaging can be performed even when the imaging fibre is moved.

[0007] Optionally, the method comprises reconstructing one or more images associated with predicted input distal image data based on proximal output image data, thereby to image one or more regions at the distal end of the imaging probe. This means that real-time imagingcan be performed with a high degree of accuracy without accessing the distal end of an imaging probe in order to recalibrate the imaging probe.

[0008] There is also provided a computer device configured to perform the method and a computer readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method.

[0009] There is also provided an imaging system comprising: a computing device; an imaging probe; a source of electromagnetic radiation and a detector for detecting electromagnetic radiation from the source at each of the imaging probe, wherein the imaging system is configured to perform the method.

[0010] There is also provided a machine learning model for predicting distal input image data, wherein the machine learning model is pre-trained machine based on correlated proximal output image data associated with a proximal end of an imaging probe with distal input image data associated with a distal end of the imaging probe, wherein the machine learning model is configured to: predict distal input image data associated with the distal end of an imaging probe based on the proximal output image data determined at the proximal end of the imaging probe; and update to provide an updated machine learning model based on the predicted input distal image data and the determined proximal output image data.

[0011] Further aspects of the invention will be apparent from the description and the appended claims.DETAILED DESCRIPTION OF EMBODIMENTS

[0012] A detailed description of embodiments is described, by way of example only, with reference to the figures in which:Figure 1 shows a schematic of an imaging apparatus;Figure 2 shows a flowchart of a method of operating an imaging apparatus;Figure 3 shows a flowchart of a method for training a machine learning model;Figure 4 shows an application of the imaging apparatus of Figure 1;Figure 5A is a graph showing the change in correlation of outputs from a multimode fibre as a function of time; andFigure 5B is a graph showing imaging reconstruction accuracy as a function of time. In the various drawings, like parts are denoted by like references.

[0013] As described above, single multimode optical fibres have properties that make them highly desirable for use as imaging probes in the field of clinical endoscopy. Imaging with such single multimode optical fibres is performed based on the pre-calibrated optical transmission properties of the optical fibres. These pre-calibrated properties are used to reconstruct images at a proximal end of the optical fibre so that imaging of a region at the distal end of the optical fibre can be performed. However, when the optical fibres are moved, for example by being flexed, changes in their optical transmission properties mean that it is no longer possible to reconstruct images without recalibration of the optical transmission properties of the optical fibres. If access to the distal end of an optical fibre is not possible, recalibration cannot be performed. This restricts the imaging capability of such imaging probes, rendering them unsuitable for endoscopic procedures where recalibration cannot be performed in situ.

[0014] The inventors have surprisingly found that this flexibility challenge may be addressed through the application of machine learning to achieve real-time fibre transmission property recalibration without requiring access to the distal end of the fibre.

[0015] Figure 1 shows a schematic of an imaging apparatus 100 with an imaging probe that can be recalibrated in real-time without requiring access to the distal end of the imaging probe. There is shown an imaging probe 18 that has a proximal end 16 and a distal end 20. In an implementation the imaging probe 18 is an optical fibre. In some implementations, the imaging probe 18 comprises a single multi-mode optical fibre. In further implementations, additionally or alternatively, the imaging probe 18 comprises a plurality of single mode fibres.

[0016] In an implementation, the imaging apparatus 100 comprises an endoscope. For example, the imaging probe 18 may form at least part of an endoscope, thereby enabling endoscopic imaging to be performed. In further implementations, the imaging apparatus 100 comprises any appropriate imaging probe that may implement the techniques described herein

[0017] In order to train a machine learning algorithm, to use a machine algorithm to predict data, and / or to image with the imaging apparatus 100, electromagnetic radiation is coupled the the imaging probe 18 at the proximal end 16 via coupling optics 14. A source of electromagnetic radiation 2 is provided and generates electromagnetic radiation that may be directed through an optical fibre 4 to coupling optics 6. In an implementation, the source of electromagnetic radiation 2 is a laser source generating light with a visible wavelength. In further implementations, the source of electromagnetic radiation 2 is any suitable source for generating electromagnetic radiation at any of one or more appropriate wavelengths for performing the methods described herein. For example, in some implementations, the imaging apparatus 100 is configured to perform fluorescence or two-photon fluorescence imaging and the source of electromagnetic radiation 2 is chosen accordingly. Whilst an optical fibre 4 maybe used to direct electromagnetic radiation from the source of electromagnetic radiation 2 to the coupling optics 6, in further implementations any suitable medium and / or components may be used to direct the electromagnetic radiation between the entities.

[0018] The coupling optics 6 enable modulation of the electromagnetic radiation by directing the electromagnetic radiation to a wavefront shaping device 8 in a manner that facilitates precise control of the electromagnetic radiation that is output by the wavefront shaping device 8. In one implementation, the coupling optics 6 comprise lenses configured to control the spatial distribution of light intensity incident on the wavefront shaping device 8. For example, the coupling optics may comprise mirrors and lenses to direct and expand the incident beam of electromagnetic radiation, thereby enabling highly controlled modulation of the light field by the wavefront shaping device 8. In further implementations, the coupling optics 6 comprise any suitable components to provide the functionality described herein.

[0019] In one implementation, the wavefront shaping device 8 is a digital micromirror device. Digital micromirror devices comprise a large number (e.g., of the order of 10000) small mirrors that can be individually operated at a high frequency (e.g., 250 Hz) that allows for modulation of light to control a wavefront pattern. A digital micromirror device provides an array of mirrors that can be used to control light intensity across a corresponding spatially distributed array of light intensities. The spatially distributed array of light intensities may be modulated by controlling one or more individual mirrors of the array to provide a plurality of modulation patterns. The light intensity distribution of intensities may be mapped to pixels such that input light intensity can be compared with output light intensity in a controlled manner. In further implementations, the wavefront shaping device 8 is any suitable electrically or optically addressed spatial light modulator (SLM), such as a liquid crystal SLM. In further implementations, the wavefront shaping device 8 is any suitable device for modulating electromagnetic radiation in accordance with the methods described herein.

[0020] Electromagnetic radiation modulated by the wavefront shaping device 8 is directed along a path 10 from the wavefront shaping device 8 to a dichroic mirror 12. The path 10 may comprise any suitable medium and / or components. The dichroic mirror 12 is arranged such that modulated electromagnetic radiation is directed along a path 11 to the coupling optics 14 and is coupled to the imaging probe 18 at the proximal end 16 of the imaging probe 18. The path 11 to the coupling optics 14 comprises any suitable medium and / or components. Whilst a dichroic mirror 12 is used to control optical paths to and from the imaging probe 18, in further implementations any suitable component may be used to provide the functionality. The coupling optics 14 comprises lenses configured to direct the modulated electromagnetic radiation into the proximal end 16 of the imaging probe 18. In further implementations, thecoupling optics 14 comprise any suitable components to provide the functionality described herein.

[0021] Electromagnetic radiation that is coupled into the imaging probe 18 is directed along the imaging probe 18 to the distal end 20 of the imaging probe 20, where the electromagnetic radiation illuminates an area at the distal end 20 of the imaging probe 18. In an implementation, such illumination is in the form of distal speckle. Illumination of a region to be imaged by the distal speckle at the distal end 20 of the imaging probe 18 results in electromagnetic radiation being reflected through the imaging probe 18 from the distal end 20 to the proximal end 16. The electromagnetic radiation that is reflected by a region at the distal end 20 of the imaging probe is directed by the imaging probe 18 from the distal end 20 to the proximal end 16 is coupled back along the path 11 between the coupling optics 14 and the dichroic mirror 12 and passes along a path 13 to further coupling optics 22. The dichroic mirror 12 serves the purpose of enabling electromagnetic radiation to be coupled into the imaging probe 18 and out of the imaging probe 18 to a detector 24. However, in further implementations any suitable one or more additional and / or alternative components are used to provide the functionality.

[0022] The further coupling optics 22 couple the reflected electromagnetic radiation from the imaging probe 18 into the detector 24. In an implementation, the further coupling optics 22 comprise one or more lenses. In further implementations, the further coupling optics 22 comprise any suitable additional and / or alternative components that enable reflected electromagnetic radiation from the imaging probe 18 to be coupled into the detector 24. In an implementation, the reflected electromagnetic radiation from the imaging probe 18 is in the form of proximal speckle. The proximal speckle detected from the proximal end 16 of the imaging probe 18 provides proximal output image data. The proximal output image data detected at the detector 24 changes depending on illumination of a region to be imaged by electromagnetic radiation from the distal end 20 of the imaging probe 18. Changes in the region to be imaged and / or the distal speckle result in changes in the detected proximal speckle and hence proximal output image data. As described further herein, control of the coupled electromagnetic radiation into an imaging probe and coordinated detection of the proximal output image data enables re -calibration and image reconstruction so that an imaging probe can be used to provide real-time imaging. Such control and coordinated detection may be performed through the use of one or more computing devices as illustrated at Figure 1.

[0023] The imaging apparatus 100 comprises a computing device 26 having a processor 28 and a memory 30. The computing device 26 is in communication with a display 32 via a communication path 27. The computing device 26 may comprise one or more inputs. In oneimplementation, the one or more inputs are provided by a user interface of the display 32. The computing device 26 is in communication with the source of electromagnetic radiation 2, wavefront shaping device 8 and / or detector 24 via one or more interfaces, thereby enabling control of the illumination of an imaging region at the distal end 20 of the imaging probe 18 and detection of reflected electromagnetic radiation from the imaging region at the distal end 20 of the imaging probe 18. Accordingly, a communication path 40 is provided between the computing device 26 and the source of electromagnetic radiation 2, a further communication path 36 is also provided between the computing device 26 and the detector 24 and, a further communication path 46 is also provided between the computing device 26 and the wavefront shaping device 8. The communication paths 27, 36, 40, 46 may comprise wired and / or wireless links. Whilst particular communication paths are shown, it will be understood that in further implementations additional and / or alternative communication paths between elements of the imaging apparatus 100 may be provided in order to enable the functionality described herein.

[0024] The computing device 26 may be configured to store one or more machine learning models in the memory 30 of the computing device 26 in order to implement the functionality described herein. In further implementations, additionally or alternatively, the computing device 26 may be in communication with a network 34 of one or more computing devices via a communication path 42 comprising wired and / or wireless links in order to provide the functionality described herein. In such implementations, one or more machine learning models are alternatively and / or additionally stored at one or more devices forming part of a network 34. For example, the network 34 may comprise one or more servers.

[0025] In an implementation, the network 34 is directly in communication with the detector 24 via a communication path 38 and / or the source of electromagnetic radiation 2 via a communication path 38 and / or the wavefront shaping device 18 via a communication path (not shown). Alternatively, or additionally, the network 34 may be in communication with the detector 24 and / or source of electromagnetic radiation 2 and / or wavefront shaping device 8 indirectly via the computing device 26.

[0026] Whilst the computing device 26 may be used to control the illumination of a region to be imaged at the distal end 20 of the imaging probe and to coordinate detection of reflected electromagnetic radiation from the proximal end 16 of the imaging probe 18, it will be understood that in further implementations the functionality described herein may be distributed between the computing device 26 and / or one or more computing devices of the network 34 in order to enable imaging probe re -calibration and imaging probe image reconstruction.

[0027] The imaging apparatus 100 is an example of a system comprising a computing device 26, an imaging probe 18, a source of electromagnetic radiation 2 and a detector 24 fordetecting electromagnetic radiation from the source 2 at each end of the imaging probe, wherein the system is configured to perform the methods described herein. Whilst the imaging apparatus 100 of Figure 1 is described with reference to a specific arrangement of components, it will be understood that in further implementations, the imaging apparatus 100 comprises additional or alternative components to provide the functionality described herein.

[0028] Figure 2 shows a method S200 of operating an imaging apparatus, such as the imaging apparatus 100 described with reference to Figure 1. The method S200 starts at step S202, where the method is initiated. The method S200 is initiated in response to a user command, such as a command input at the computing device 26 to initiate imaging using the imaging probe 18. In further implementations, additionally or alternatively, the user command may comprise any suitable combination of automated and / or manual actions to instigate imaging using imaging apparatus, such as the imaging apparatus 100 described with reference to Figure 1.

[0029] In response to the user command, the method S200 moves to step S204, where a machine learning model is pre-trained. The machine learning model may be a model that is stored in the memory 30 of the computing device 26, or in a memory of one or more computing devices of the network 34. The machine learning model is pre-trained based on correlating proximal output image data associated with a proximal end of an imaging probe with distal input image data associated with a distal end of the imaging probe. The machine learning model may be pre-trained using the same or different apparatus that is used for imaging. A method for pre-training the machine learning model is described in further detail with reference to Figure 3. Once the machine learning model has been pre-trained, the process moves to step S206.

[0030] At step S206, proximal output image data associated with the proximal end of the imaging probe is determined. In an implementation, proximal output image data associated with the proximal end 16 of the imaging probe 18 is determined by coupling electromagnetic radiation into the imaging probe 18 and detecting the coupled electromagnetic radiation at the proximal end 16 of the imaging probe 18. The coupled electromagnetic radiation at the proximal end 16 of the imaging probe is radiation that has been directed towards the distal end 20 of the imaging probe 18 and has been reflected back along the imaging probe 18 as described above with reference to the arrangement of Figure 1. The reflected electromagnetic radiation is coupled via coupling optics 22 into a detector 24 to provide proximal output image data. In an implementation, the proximal output image data comprises one or more batches of 500 reflected proximal speckle images. In further implementations, the proximal output image comprises one or more batches of any appropriate number of proximal speckle images. The proximal outputimage data may be processed by the processor 28 of the computing device 26 and stored in the memory 30 of the computing device 26. In further implementations, alternatively or additionally, the proximal output image data is processed and / or stored across one or more computing devices and / or servers forming part of the network 34 that is in communication with the imaging apparatus 100. Once the proximal output image data associated with the proximal end of the imaging probe has been determined, the process moves to step S208.

[0031] At step S208, the pre-trained machine learning model, such as the pre-trained machine learning model described with reference to Figure 3, is used to predict distal input image data associated with the distal end of the imaging probe based on the determined proximal output image data. In an example implementation, the pre -trained machine learning model is a standard convolutional neural network (CNN) with three convolutional layers that has been trained on 10% of 40,000 collected input-output data pairs using Binary Cross Entropy cost as the loss function. In further examples, the machine learning model is any suitable model that has been appropriately trained to provide the functionality described herein. Accordingly, the proximal output image data obtained at step S206 is used as an input to the machine learning model. The machine learning model predicts distal input image data associated with the proximal output image data. Once distal input image data associated with the distal end of the imaging probe has been determined at step S208 the process moves to step S210.

[0032] At step S210, the machine learning model is updated based on the predicted input distal image data and the determined proximal output image data to provide an updated machine learning model. In one implementation, the machine learning model comprises a selfsupervised learning (SSL) algorithm that is adapted to update a machine learning model, such as a neural network, in real-time, thereby enabling reliable data transmission through multi-mode fibres. In an example implementation, the determined proximal output image data comprises one or more batches, wherein each batch comprises 500 images of reflected proximal speckle. The one or more batches of images of reflected proximal speckle are processed by the machine learning model, such as a standard CNN with three convolutional layers, to provide corresponding predicted distal input image data. The SSL algorithm processes the one or more batches of images of reflected proximal speckle and corresponding predicted distal input image data in order to refine the SSL algorithm and update the CNN accordingly. In further implementations, any suitable batch sizes of images of reflected proximal speckle are used. Optionally, the predicted input distal image data is output, for example by the display 32 of the computing device 26. Once the machine learning model has been updated based on the predicted input distal image data and the determined proximal output image data to provide an updated machine learning model, the updated machine learning model may be used to predictinput distal image based on determined proximal output image data with improved accuracy. The process may then move to step S212.

[0033] At step S212 the method S200 enters a recalibration phase of iteratively adjusting the imaging probe for one or more iterations. Adjustments to the imaging probe, such as the imaging probe 18 described with reference to Figure 1 alter the transmission properties of the imaging probe so that the transmission matrix between proximal output image data and corresponding distal input image data changes. By iteratively adjusting the imaging probe, determining proximal output image data, predicting corresponding distal input image data and updating the machine model accordingly, strong correlation between detected proximal output image data and predicted corresponding distal input image data may be maintained, thereby enabling accurate image reconstruction at locations that do not allow for conventional calibration of an imaging probe. In an implementation, iteratively adjusting the imaging probe comprises controlling an insertion device thereby to move the distal end of the imaging probe to an imaging destination in a series of discrete steps. Such controlled insertion may be performed manually, for example by an operator of the imaging apparatus. Alternatively, or additionally, controlled insertion may comprise movement in automated discrete steps, for example using one or more computer-controlled devices, such as one or more devices controlled by the computing device 26 of the imaging apparatus 100. The discrete steps may comprise moving the distal end of the imaging probe to the imaging destination at regularly spaced time intervals and / or regularly spaced positions. Alternatively, the discrete steps may comprise moving the distal end of the imaging probe to the imaging destination at time intervals and / or spaced positions based on the location of the distal end of the imaging probe with respect to the imaging destination. In further implementations, the imaging probe may be adjusted using any appropriate adjustment of a property of the imaging probe. Once the imaging probe has been adjusted at step S212, the process moves to step S214.

[0034] At step S214, further proximal output image data from the proximal end of the imaging probe is determined. The further proximal output image data from the proximal end of the imaging probe is determined in a similar manner to that described with reference to step S206. For example, in an implementation the determined proximal output image data comprises one or more batches of 500 images of reflected proximal speckle. Once further proximal output image data has been determined from the proximal end of the imaging probe, the process moves to step S216.

[0035] At step S216, the updated machine learning model that was updated at step S210 is used to predicted further distal input image data associated with the distal end of the imaging probe based on the determined further proximal output image data that was determined at stepS214. For example, in an implementation, the determined proximal output image data comprises one or more batches of 500 images of reflected proximal speckle that are processed by a pre-trained CNN with three convolutional layers to predict corresponding distal input image data. This forms part of the incremental measurement, prediction and updating of a machine learning model to enable accurate prediction of distal input image data and subsequent imaging. Once the further distal input image data associated with the distal end of the imaging probe has been predicted at step S216, the process moves to step S218.

[0036] At step S218 the updated machine learning model is further updated based on the predicted further distal input image data obtained at step S216 and the further proximal output image data obtained at step S214 in order to provide a further updated machine learning model. In one implementation, as described above, the machine learning model is updated using the self-supervised learning (SSL) algorithm that is adapted to update a machine learning model, such as a neural network, in real-time, thereby enabling reliable data transmission through multi-mode fibres. In an example implementation, the determined proximal output image data comprises one or more batches of 500 images of reflected proximal speckle. The one or more batches of images of reflected proximal speckle are processed by the machine learning model, such as by a standard CNN with three convolutional layers, to provide corresponding predicted input image data. The SSL algorithm of the machine learning model processes the one or more batches of images of reflected proximal speckle and corresponding predicted distal input image data in order further to refine the SSL algorithm and further update the CNN accordingly. In further implementations, any suitable batch sizes of images of reflected proximal speckle are used. Once the machine learning model has been further updated at step S218, the process moves to step S220.

[0037] At step S220 it is determined whether or not the imaging probe is at the imaging destination. If the imaging destination has been reached, the process moves to step S222. If the imaging destination has not been reached, the process returns to step S212, where the imaging probe is adjusted in another iteration. Subsequently, further proximal output image data from the proximal end of the imaging probe is determined at step S214, further distal input image data associated with the distal end of the imaging probe is determined based on the most recently determined further proximal output image data at step S216. The machine learning model is then updated at step S218 based on the most recently predicted further distal input image data and the most recently determined further proximal output image data to provide a further updated machine learning model that corresponds to the latest iteration of the adjustment of the imaging probe. The process then moves to step S220, where it is determined whether or not the imaging probe is at the imaging destination. If the imaging destination has been reachedthe process moves to step S222. If the imaging destination has still not been reached, the process returns once again to step S212. Steps S212 to S220 are repeated as part of the iterative adjustment of the imaging probe until the imaging probe has reached the imaging destination.

[0038] Whilst iterative adjustment is described such that the machine learning model is updated based on the most recently determined further proximal output image data and corresponding predicted distal input image data, it will be understood that the machine learning model may be updated based on determining proximal output image data and corresponding predicted distal input image data in batches, and processing sequential batches in order to update the machine learning model. For example, in one implementation, a pre-trained CNN as described herein may be used to predict distal image data corresponding to determined proximal output image data comprising a batch of 500 reflected proximal speckle images. The determined proximal output image data may be continuously captured and processed by the CNN to provide predicted distal input image data corresponding to the determined proximal output image data. The SSL algorithm may update the CNN based on a select number of most recent input-output data pairs. For example, the CNN may be updated based on the most recent 2000 input-output data pairs. In further examples, the CNN may be updated based on any appropriate number of input-output data pairs and for any appropriate batch size of determined reflected proximal speckle images. The CNN may be updated by the SSL algorithm after each adjustment of the imaging probe. Additionally, or alternatively, the CNN may be updated by the SSL algorithm at regular time intervals. Advantageously, by capturing real-time speckle images reflected by the distal end of an imaging probe, the high prediction accuracy of a CNN over short time intervals may be used to form new input-output data training pairs for iterative updates of the CNN.

[0039] Recalibration of the imaging apparatus described with reference to Figure 2 comprises iteratively adjusting the imaging probe, determining further proximal output image data, predicting further distal input image data and updating the machine learning model accordingly. In some implementations, recalibration may be performed offline. For example, the further proximal output image data from the proximal end of the imaging probe for each iteration of the iterative adjustment may be stored. In an example of an implementation, the further proximal output image data may be stored in the memory 30 of the computing device 26. Alternatively, or additionally, the further proximal output image data may be stored at one or more computing devices of the network 34. Once the further proximal output image data has been stored, the updated machine learning model may subsequently be used to predict further distal input image data associated with the distal end of the imaging probe based on the stored further proximal output image data. The updated machine learning model can then be furtherupdated based on the predicted further distal input image data and the further proximal output image data to provide the further updated machine learning model.

[0040] At step S222, imaging may be performed using the imaging probe. Imaging may be performed based on the predicted distal input image data. In an implementation, imaging is performed by reconstructing one or more images associated with predicted distal image data based on proximal output image data, thereby to image one or more regions at the distal end of the imaging probe. In an implementation, image reconstruction may be performed by applying a predetermined algorithm that is configured to receive predicted distal input image data and reconstruct an image based on the predicted distal input image data. In further implementations, image reconstruction is performed by applying any suitable alternative and / or additional steps.

[0041] In order to image, a proximal output image associated with the proximal end of the imaging probe may be determined by coupling electromagnetic radiation into the imaging probe and detecting the electromagnetic radiation reflected by a region at the distal end of the imaging probe at a detector coupled with the proximal end of the imaging probe. In order to image, the coupled electromagnetic radiation that is coupled into the imaging probe may be modulated to form a focal spot at the distal end of the imaging probe. In an example of an implementation, the electromagnetic radiation may be modulated by controlling a digital micromirror device, thereby to form the focal spot. In further implementations, the electromagnetic radiation may be modulated by controlling any suitable alternative and / or additional wavefront shaping device. For example, an electrically or optically addressed spatial light modulator, such as a liquid crystal SLM may be used. The way in which the focal spot is formed at the distal end of the imaging probe may be controlled, so that the relative position of the focal spot moves. In an implementation, the electromagnetic radiation that is coupled into the imaging probe is modulated in order to rasterize the focal spot over one or more predetermined regions at the distal end of the imaging probe. Rasterizing the focal spot over one or more predetermined regions at the distal end of the imaging probe enables image data to be collected that can be used to perform image reconstruction. This can be achieved by predicting distal input image data based on the determined proximal output image data. As the focal spot is rasterized over the one or more predetermined regions, one or more images may be formed based on the predicted distal input image data. This enables real-time imaging of the one or more predetermined regions at the distal end of the imaging probe. In further implementations, imaging is performed by any appropriate control of the electromagnetic radiation coupled to the imaging probe and subsequent detection of the proximal output image data.

[0042] Once imaging has finished at step S222, the process ends at step S224.Alternatively, once imaging at the imaging destination has been completed at step S222, theprocess may return to step S212, where the imaging probe is adjusted and the process of iterative adjustment of the imaging probe to image at a further destination may be performed.

[0043] Whilst the method S200 is described with reference to the imaging apparatus 100 of Figure 1, it will be understood that the method may be applied to any suitably arranged imaging apparatus to provide the functionality described herein.

[0044] Whilst the method S200 of Figure 2 is described with reference to steps S202 to S224 in a particular order, it will be understood that the steps may be performed in any appropriate order in order to implement the functionality described herein. In further implementations, the method S200 is performed with any appropriate additional and / or alternative steps in order to provide the functionality described herein.

[0045] At least some of the steps of the method S200 may be stored as instructions of a computer readable medium such that, when executed by a computer, the instructions cause the computer to perform at least some step of the method S200 described herein.

[0046] Figure 3 is a flowchart of a method S300 of pre-training a machine learning model, such as the machine learning model described with reference to the method S200 of Figure 2. The machine learning model is a neural network, such as a conventional neural network (CNN). In an example implementation, a standard CNN with three convolutional layers is used. In further implementations, a standard CNN with any appropriate number of convolutional layers may be used. In further implementations, the machine learning model may be any appropriate machine learning model that provides the functionality described herein.

[0047] In order to pre-train a machine learning model, training data is generated. Training data is generated for an imaging probe, such as the imaging probe 18 described with reference to Figure 1. The method S300 may be used to pre-train a machine learning model that is stored in the memory 30 of the computing device 26 described with reference to the imaging apparatus 100 of Figure 1. Alternatively, or additionally, the method S300 may be used to pre-train a machine learning model that is stored at any accessible location to provide the functionality described herein. The process starts at step S302 in response to a user command. The user command may comprise any appropriate combination of automated and / or manual inputs. For example, a user may input a command at the computing device 26 to run one or more processes for generating training data to pre-train a machine learning model. Once instigated, the process moves to step S304.

[0048] At step S304 electromagnetic radiation is coupled into the imaging probe. In the implementation described with reference to Figure 1, electromagnetic radiation is coupled into the imaging probe 18 from the source of electromagnetic radiation 2. The electromagnetic radiation is controlled by coupling optics 6 in order to illuminate a wavefront shaping device 8.In one implementation, the source of electromagnetic radiation 2 is a laser and the wavefront shaping device 8 is a digital micromirror device. The wavefront shaping device 8 is controlled in order to provide modulation patterns that are directed along the pathway 10 to a dichroic mirror 12 and subsequently along a further pathway 11 where the modulation patterns are coupled to the imaging probe 18 at the proximal end 16 of the imaging probe, via appropriate coupling optics 14. In further implementations, the electromagnetic radiation is coupled into the imaging probe using any suitable means to implement the method described herein. Once the electromagnetic radiation has been coupled into the imaging probe at step S304, the process moves to step S306.

[0049] At step S306 the electromagnetic radiation coupled into the imaging probe at step S304 is detected at each of the proximal end and the distal end of the imaging probe. A detector, such as the detector 24 described with reference to Figure 1 may be used to detect electromagnetic radiation from the proximal end 16 of the imaging probe 18 as described above with reference to Figure 1. An additional detector may be used at the distal end 20 of the imaging probe 18 in order to detect electromagnetic radiation from the distal end 20 in an analogous manner. Coupling optics may be used at the distal end 20 of the imaging probe 18 in order to couple electromagnetic radiation from the distal end 20 of the imaging probe 18 into a detector in an analogous manner as described with reference to electromagnetic radiation coupled from the proximal end 16 of the imaging probe 18. In further implementations, electromagnetic radiation from the proximal end of the imaging probe is coupled to a detector in any appropriate manner. Once the electromagnetic radiation has been detected at each of the proximal end and the distal end of the imaging probe, input and output data is provided that may be used to train a machine learning model to correlate proximal and distal image data. In an implementation, a transmission matrix associated with proximal output image data and corresponding distal input image data may be determined. In one implementation, in order to determine a transmission matrix associated with proximal output image data and corresponding distal the process moves to step S308, where the electromagnetic radiation that is coupled into the imaging probe is modulated to form a plurality of modulation patterns. In an implementation, the electromagnetic radiation is modulated to form a plurality of modulation patterns and each modulation pattern of the plurality of modulation patterns is correlated with a respective transmission matrix associated with proximal output image data and corresponding distal input image data. The plurality of modulation patterns may be formed by modulating the electromagnetic radiation by controlling a wavefront shaping device thereby to form the plurality of modulation patterns. For example, in an implementation described with reference to Figure 1, the computing device 26 is configured to control the wavefront shaping device 8 inorder to change the spatial distribution of radiation intensities. As described with reference to Figure 1, the wavefront shaping device may be a digital micromirror device, or liquid crystal SLM, or any suitable additional and / or alternative component.

[0050] For each modulation pattern, the electromagnetic radiation coupled into the imaging probe may be detected at each of the proximal end 16 and the distal end 20 of the imaging probe 18. As described herein, the proximal output image data from the proximal end 16 of the imaging probe 18 may be determined by detecting reflected proximal speckle corresponding to proximal output image data at a detector at the proximal end of the imaging probe and distal input image data may be determined by detecting distal speckle at a detector at the distal end of the imaging probe. The proximal output image data, distal input image data and modulation pattern data may be recorded by the computing device 26 and stored in the memory 30. Through the control of the plurality of modulation patterns by the computer device 26, the computing device 26 can determine proximal output image data from the proximal end 16 of the imaging probe 18 and distal input image data from the distal end 20 of the imaging probe. Using this data, a machine learning model may be trained such each modulation pattern of the plurality of modulation patterns may be correlated with a respective transmission matrix associated with proximal output image data and corresponding distal input image data.

[0051] In order to provide a sufficiently accurate machine learning model, sufficient data must be collected. In an example of an implementation, 40000 input-output data pairs of distal input image data and corresponding proximal output image data are collected. A proportion, for example, 10%, of the input-output data pairs (i.e., 4000 input-output data pairs) is used to train a machine learning model, such as a standard CNN with three convolutional layers. In further examples, any suitable number of input-output pairs of data are collected and any appropriate proportion of the input-output data pairs is used to train the machine learning model. Whilst transmission matrices associated with proximal output image data and corresponding distal input image data may be determined by modulating the electromagnetic radiation to form a plurality of modulation patterns and correlating each modulation pattern of the plurality of modulation patterns with a respective transmission matrix associated with proximal output image data and corresponding distal input image data, in further implementations, the transmission matrices are determined using any suitable additional and / or alternative steps. Once sufficient training data has been obtained the process moves to step S310.

[0052] At step S310, the machine learning model is pre-trained based on the proximal output image data and the corresponding distal input image data so that each modulation pattern of the plurality of modulation patterns is correlated with a respective transmission matrix associated with proximal output image data and corresponding distal input image data. Therespective transmission matrix enables the intensity output at a coordinate of the proximal output image data to be mapped to a corresponding coordinate of the distal input image data. In one implementation the machine learning model is a neural network, such as a CNN. In an example, the machine learning model is a standard CNN with three convolutional layers. In further examples, the CNN comprises any suitable number of convolutional layers. In further implementations, the machine learning model comprises any suitable alternative and / or additional machine learning model. The machine learning model is trained using any appropriate technique. In one implementation, the machine learning model is trained using Binary Cross Entropy cost as the loss function. In further implementations, the machine learning model is trained using any appropriate loss function and / or mechanism. Once sufficiently trained, the machine learning model is able to predict, with a predetermined accuracy, distal input image data based on proximal output image data. This means the functionality of a transmission matrix is provided without the need to perform explicit calculations based on the properties of the imaging probe.

[0053] In an implementation, the machine learning model is trained at a computing device, such as the computing device 26 of the imaging apparatus 100. However, it will be understood that in further implementations the machine learning model may be trained at any appropriate computing device by storing the data obtained at step S308 and training the machine learning model using the stored data.

[0054] Following pre-training of the machine learning model, the process moves to step S312 where the pre-trained machine learning model is output. As described above, in one implementation, the pre-trained machine learning model is a standard CNN with three convolutional layers that has been trained on 4000 input-output data pairs using a Binary Cross Entropy cost such that the trained CNN can be used to predict distal input image data where there is no access to the distal end of an imaging probe. In one implementation, outputting the machine learning model comprises storing the machine learning model in the memory 30 of the computing device 26 in order to perform further predictions. In further implementations, the pre-trained machine learning model is output and / or stored at any appropriate location for further use. The machine learning model is a model for predicting distal input image data that has been pre-trained based on correlated proximal output image data associated with a proximal end of an imaging probe with distal input image data associated with a distal end of the imaging probe. The machine learning model is configured to predicted distal input image data associated with the distal end of an imaging probe based on the proximal output image data determined at the proximal end of the imaging probe. The machine learning model is configured to update to provide an updated machine learning model based on the predicted inputdistal image data and the determined proximal output image data. Accordingly, the machine learning model that is output may be used as an initial model that may be updated as part of a recalibration process described herein.

[0055] Whilst the method S300 of Figure 3 is described with reference to the imaging apparatus 100 of Figure 1, it will be appreciated that the method S300 may be implemented at any suitable apparatus in order to generated a pre-trained machined learning model. Whilst the method S300 of Figure 3 is described with reference to steps S302 to S312 in a particular order, it will be understood that the steps may be performed in any appropriate order in order to implement the functionality described herein. In further implementations, the method S300 is performed with any appropriate additional and / or alternative steps in order to provide the functionality described herein.

[0056] At least some of the steps of the method S300 may be stored as instructions of a computer readable medium such that, when executed by a computer, the instructions cause the computer to perform at least some step of the method S300 described herein.

[0057] Figure 4 shows an application 400 of the imaging apparatus 100 of Figure 1. There is shown an imaging probe 18 that is coupled through optical couplers 14, 22 such that light from a digital micromirror device 8 is directed along a path to the imaging probe 18 and reflected light from the distal end of the imaging probe is detected along a further path 13, in the manner described with reference to Figure 1. The application 400 implements the pre-trained machine learning model described herein so that distal end input image data 406 is correlated with proximal output image data 408. The imaging probe 18 is then inserted in an arbitrary duct 402 such that the pre-trained machine learning model is iteratively updated based on predicted distal input image data 410 and detected proximal output image data 408 until the imaging probe 18 reaches the destination of the region to be imaged 404. Once at the region to be imaged 404, imaging may be performed using the imaging probe 18 such that distal input image data 410 is predicted based on detected proximal output image data 408.

[0058] Figures 5A and 5B demonstrate how recalibration of an imaging probe described herein enables high accuracy reconstruction of images using an example of an imaging probe described with reference to Figure 1.

[0059] Figure 5A is a graph 500A showing the change in correlation of outputs from a multimode fibre as a function of time. In the experimental example of Figure 5 A, the multimode fibre was a stationary 40 pm diameter, 100 m long fibre. Correlation on the y-axis 504A is plotted against time, in seconds, on the x-axis 502A. A first trace 506A shows the change in correlation as a function of time between outputs at time = t and time = 0, for the same input. In the example of Figure 5A, the correlation drops to around 80% after 80 seconds.However, this change is gradual, as illustrated by a second trace 508A. The second trace 508A shows the change in correlation as a function of time between temporally adjacent outputs at time = t and time = t-1. This demonstrates the high similarity between adjacent states in the time domain. This property means that it is possible to use a pre-trained machine learning model when the distal end access is available to obtained the next unknown distal end information immediately after an endoscopic procedure starts when the distal end access is unavailable for recalibration.

[0060] Figure 5B is a graph 500B showing imaging reconstruction accuracy as a function of time using a pre-trained neural network to predict distal input image data from an imaging probe based on detected proximal output image data. Reconstruction accuracy as a percentage is plotted on the y-axis 504B as a function of time, in seconds, on the x-axis 502B. A first trace 506B shows results in the case where re -calibration of the imaging apparatus is not performed. In this case, the machine learning model used to predict the distal image input data is not updated and the reconstruction accuracy drops significantly as a function of time. A second trace 508B shows results in the case where re -calibration of the imaging apparatus is performed such that the machine learning model that is used to predict distal input image data is updated in accordance with the method described herein. As shown at Figure 5B, with re -calibration, a reconstruction accuracy of close to 100% is maintained as a function of time.

[0061] The apparatus and methods described herein provide a way of enabling real-time imaging with imaging probes that can be used in situations where there is no access to the distal end of the imaging probe. Whilst particular components and steps are described, it will be understood that additional and / or alternative components and steps may be used to implement the functionality described herein. For example, in implementations where the imaging probe comprises a plurality of single mode fibres, proximal output image data and distal input image data may correspond to a pixelated spatial distribution of intensities of electromagnetic radiation that are associated with each fibre of the plurality of single mode fibres. This may contrast to proximal and distal speckle data of a single multi-mode fibre. However, the principles described herein may be applied to enable real-time recalibration and imaging in both cases.

[0062] The methods of the present invention may be performed by computer systems comprising one or more computers. A computer used to implement the invention may comprise one or more processors, including general purpose CPUs, graphical processing units (GPUs), tensor processing units (TPU) or other specialised processors. A computer used to implement the invention may be physical or virtual. A computer used to implement the invention may be a server, a client or a workstation. Multiple computers used to implement the invention may be distributed and interconnected via a network such as a local area network (UAN) or wide areanetwork (WAN). Individual steps of the method may be carried out by a computer system but not necessarily the same computer system. Results of a method of the invention may be displayed to a user or stored in any suitable storage medium. The present invention may be embodied in a non-transitory computer-readable storage medium that stores instructions to carry out a method of the invention. The present invention may be embodied in a computer system comprising one or more processors and memory or storage storing instructions to carry out a method of the invention. The present invention may be incorporated into a medical imaging device or into software updates or add-ons for such a device.

[0063] Having described the invention it will be appreciated that variations may be made to the above described embodiments, which are not intended to be limiting. The invention is defined in the appended claims and their equivalents.

Claims

CLAIMS1. A method of operating an imaging apparatus, the method comprising:pre-training a machine learning model based on correlating proximal output image data associated with a proximal end of an imaging probe with distal input image data associated with a distal end of the imaging probe;determining proximal output image data associated with the proximal end of the imaging probe;using the pre-trained machine learning model to predict distal input image data associated with the distal end of the imaging probe based on the determined proximal output image data; andupdating the machine learning model based on the predicted input distal image data and the determined proximal output image data to provide an updated machine learning model.

2. The method according to claim 1, comprising:iteratively adjusting the imaging probe and for each of one or more iterations: determining further proximal output image data from the proximal end of the imaging probe;using the updated machine learning model to predict further distal input image data associated with the distal end of the imaging probe based on the determined further proximal output image data; andupdating the updated machine learning model based on the predicted further distal input image data and the determined further proximal output image data to provide a further updated machine learning model.

3. The method according to any preceding claim, wherein pre-training the machine learning model comprises:coupling electromagnetic radiation into the imaging probe; anddetecting the coupled electromagnetic radiation at each of the proximal end and the distal end of the imaging probe.

4. The method according to claim 3, further comprising:determining a transmission matrix associated with proximal output image data and corresponding distal input image data.

5. The method according to claim 4, wherein coupling electromagnetic radiation into the fibre comprises:modulating the electromagnetic radiation to form a plurality of modulation patterns; andcorrelating each modulation pattern of the plurality of modulation patterns with a respective transmission matrix associated with proximal output image data and corresponding distal input image data 5, optionally wherein modulating the electromagnetic radiation comprises controlling a wavefront shaping device thereby to form the plurality of modulation patterns, optionally wherein the wavefront shaping device is a digital micromirror device or spatial light modulator.

6. The method according to any preceding claim, wherein pre-training the machine learning model comprises:detecting reflected proximal speckle corresponding to proximal output image data at a detector at the proximal end of the imaging probe; anddetecting distal speckle at a detector the distal end of the imaging probe corresponding to distal input image data.

7. The method according to any preceding claim, comprising:reconstructing one or more images associated with predicted input distal image data based on proximal output image data, thereby to image one or more regions at the distal end of the imaging probe.

8. The method according to any preceding claim, wherein determining proximal output image data associated with the proximal end of the imaging probe comprises:coupling electromagnetic radiation into the imaging probe; anddetecting the coupled electromagnetic radiation at a detector coupled with the proximal end of the imaging probe.

9. The method according to claim 8, wherein coupling the electromagnetic radiation into the imaging probe comprises:modulating the electromagnetic radiation to form a focal spot at the distal end of the imaging probe, optionally wherein modulating the electromagnetic radiation comprises controlling a digital micromirror device, thereby to form the focal spot.

10. The method according to claim 9, wherein modulating the electromagnetic radiation comprises rasterizing the focal spot over one or more predetermined regions at the distal end of the imaging probe.

11. The method according to claim 10, comprising forming one or more images based on the predicted distal input image data corresponding to the rasterized focal spot, thereby to enable real-time imaging of the one or more predetermined regions at the distal end of the imaging probe.

12. The method according to any preceding claim, wherein the machine learning model is a neural network, preferably wherein the neural network is a convolutional neural network.

13. The method according to any preceding claim, wherein the imaging probe comprises a single multi-mode optical fibre and / or wherein the imaging probe comprises a plurality of single mode fibres.

14. The method according to any of claims 3 to 13, wherein coupling electromagnetic radiation to the fibre comprises directing light from a laser source to the imaging probe.

15. The method according to any preceding claim, wherein the imaging apparatus comprises an endoscope.

16. The method according to of claims 2 to 15, wherein iteratively adjusting the imaging probe comprises controlling an insertion device thereby to move the distal end of the imaging probe to an imaging destination in a series of discrete steps.

17. The method according to claim 16, wherein the discrete steps comprise moving the distal end of the imaging probe to the imaging destination at regularly spaced time intervals and / or regularly spaced positions or wherein the discrete steps comprise moving the distal end of the imaging probe to the imaging destination at time intervals and / or spaced positions based on the location of the distal end of the imaging probe with respect to the imaging destination.

18. The method according to any of claims 2 to 17, comprising:storing the determined further proximal output image data from the proximal end of the imaging probe for each iteration of the iterative adjustment; andsubsequently using the updated machine learning model to predict further distal input image data associated with the distal end of the imaging probe based on the stored further proximal output image data; andupdating the updated machine learning model based on the predicted further distal input image data and the further proximal output image data to provide the further updated machine learning model.

19. The method according to any preceding claim, wherein the imaging probe is an optical fibre.

20. A computing device configured to perform the method of any of claims 1 to 19.

21. A computer readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any of claims 1 to 19.

22. An imaging system comprising:a computing device;an imaging probe;a source of electromagnetic radiation; anda detector for detecting electromagnetic radiation from the source at each end of the imaging probe, wherein the imaging system is configured to perform the method of any of claims 1 to 19.

23. The imaging apparatus according to claim 22, comprising an endoscope and / or wherein the imaging apparatus is configured to perform fluorescence or two-photon fluorescence imaging.

24. A machine learning model for predicting distal input image data, wherein the machine learning model is pre-trained machine based on correlated proximal output image data associated with a proximal end of an imaging probe with distal input image data associated with a distal end of the imaging probe, wherein the machine learning model is configured to:predict distal input image data associated with the distal end of an imaging probe based on the proximal output image data determined at the proximal end of the imaging probe; andupdate to provide an updated machine learning model based on the predicted input distal image data and the determined proximal output image data.

25. The machine learning model according to claim 24, wherein the machine learning model is a neural network, preferably wherein the neural network is a convolutional neural network.

Images