Oral probe device and system

Abstract

An oral probe device for insertion into a user's mouth is described. The oral probe device comprises an image sensor, a broadband light source for illuminating a field of view of the image sensor, and a narrowband light source for stimulating fluorescence in the user's 5 mouth. The oral probe device further comprises a controller configured to control operation of the broadband light source and the narrowband light source, and to capture image data with the image sensor.

Description

P005261-W0011ORAL PROBE DEVICE AND SYSTEMBACKGROUND

[0001] Good oral hygiene relies on regular cleaning inside the mouth, including brushing the teeth and caring for the gums and tongue. Such cleaning activities can be complemented by inspection of the mouth, to ensure that cleaning is being performed adequately and identify areas in the mouth which may require further attention. Inspection of the mouth may be performed using various techniques, such as visual inspection by a dentist, and / or using an imaging device that is configured to capture images inside the mouth.SUMMARY

[0002] In a first aspect, the invention provides an oral probe device for insertion into a user’s mouth, the oral probe device comprising: an image sensor; a broadband light source for illuminating a field of view of the image sensor; a narrowband light source for stimulating fluorescence in the user’s mouth; and a controller configured to control operation of the broadband light source and the narrowband light source, and to capture image data with the image sensor.

[0003] In this manner, the oral probe device enables images of the user’s mouth to be captured under different illumination conditions. For example, the controller can activate the broadband light source to capture a broadband (e.g. white light) image of the user’s mouth. Alternatively, the controller can activate the narrowband light source, for instance to capture fluorescence images in the user’s mouth, e.g. to enable plaque or dentine detection.

[0004] When the broadband light source is used for capturing an image, this may provide a clear image of objects in the user’s mouth, such as teeth, gums, tongue, etc. Moreover, images captured with the broadband light source will have a more ‘natural’ colour profile, enabling a user to more easily observe and distinguish features in the captured images. On the other hand, images captured with the narrowband light source may include areas of fluorescence (e.g. corresponding to plaque and / or dentine in the mouth), and may have a less ‘natural’ colour profile which may make features in the mouth less clearly visible. By enabling the oral probeP005261-W0012device to capture images with both the broadband and narrowband light sources, it is possible to locate fluorescent features observed with the narrowband light source with respect to objects in the mouth observed with the broadband light source. In particular, as the same image sensor is used for capturing images with both the broadband and the narrowband light sources, images captured with the two light sources can be accurately registered with respect to one another, thus facilitating locating features of interest in the mouth. For example, the location of fluorescent features detected in images captured with the narrowband light source could be indicated in images captured with the broadband light source, to provide a visualisation of the location of the features in the mouth. As another example, the controller could switch back and forth between the broadband light source and the narrowband light source, to enable a user to determine the location of fluorescent features with respect to objects observed in the broadband images. Accordingly, the oral probe device facilitates inspecting and locating fluorescent features in a user’s mouth.

[0005] The oral probe device may comprise a handle (or handpiece), and a probe portion which is insertable into the user’s mouth. Thus, in use, the probe portion may be inserted into the mouth, whilst the handle may remain outside the mouth. The probe portion may, for example, comprise an elongate shaft that extends from the handle. The image sensor and the broadband and narrowband light sources may be coupled to the probe portion, to enable illumination and imaging of the mouth when the probe portion is inserted into the mouth. For example, the image sensor and / or the light source may be disposed (mounted) on the probe portion, e.g. near a distal end of the probe portion.

[0006] Alternatively, the image sensor and the light source may be located in the handle, with a respective light guide arranged to optically couple the image sensor and light sources to a distal end of the oral probe device. For example, a fibre optic light guide or a light pipe may be used for this purpose. The light guide may be arranged to transmit light from broadband light source and the narrowband light source in the handle to the distal end of the probe, where it can illuminate the mouth. Similarly, a light guide may be arranged to receive light from the mouth and direct the received light to the image sensor. The light guide could be made of fiber opticsP005261-W0013or transparent plastic, designed to bend or curve along a length of the oral probe device to deliver and collect light at the probe's tip. Positioning the image sensor and the light source in the handle may improve heat dissipation, and allow for more flexibility in the design of the probe portion, e.g. enabling a reduction in size which may improve ergonomics, while still providing effective illumination and imaging of the mouth.

[0007] The image sensor may comprise any suitable image sensor, such as a camera (digital camera). The image sensor may comprise an array of photosensitive elements (e.g. pixels). For example, the image sensor may comprise a charge-coupled device (CCD) image sensor, or a complementary metal-oxide-semiconductor (CMOS) image sensor.

[0008] The broadband light source and the narrowband light source may each comprise any suitable type of light source, such as a light-emitting diode (LED).

[0009] The broadband light source may be configured to emit light across a wavelength band (i.e. range of wavelengths) in the visible spectrum. For example, the broadband light source may be a white light source, e.g. a white light LED. The broadband light source may be configured to emit light across the entire visible spectrum, e.g. from about 400 nm to 750 nm. By way of example, a typical white LED will emit light in the range of about 400 nm to 750 nm or 800 nm.

[0010] The narrowband light source is configured to emit light at a wavelength for stimulating fluorescence in the user’s mouth. Accordingly, the wavelength of the narrowband light source may be selected to cause fluorescence in features of interest in the mouth. For instance, ultraviolet light and / or blue light may cause features such as plaque and dentine in the mouth to fluoresce. Thus, the narrowband light source may comprise an ultraviolet light source or a blue light source.

[0011] A wavelength band (or spectrum) emitted by the narrowband light source is narrow compared to a wavelength band emitted by the broadband light source. For example, the narrowband light source may be configured to only emit light within a relatively narrow range about a central (or peak) wavelength of the narrowband light source. By way of example, the narrowband light source may have a spectral halfwidth (i.e. full-width at half-maximum) of less than 40 nm. In some cases, the spectral half-width may be less than 30 nm, less than 20 nm, or less than 15 nm.P005261-W0014

[0012] The broadband light source is arranged to illuminate the field of view of the image sensor. In other words, an area imaged by the image sensor is illuminated by the broadband light source. Thus, objects imaged by the image sensor may be illuminated with the broadband light source. For example, the broadband light source may be located next to the image sensor, such that light emitted by the broadband light source is directed into the area imaged by the image sensor.

[0013] The narrowband light source may similarly be arranged to emit light into the field of view of the image sensor. In this manner, fluorescent features within the field of view can be imaged by the image sensor. The narrowband light source may be located next to the image sensor and / or to the broadband light source.

[0014] The controller may comprise any suitable processing device or system that is configured to perform the control operations described herein. For example, the controller may be in the form of a microcontroller in the oral probe device.

[0015] Control of the broadband light source by the controller may comprise activating or deactivating (switching on or off) the broadband light source, and / or controlling a brightness of the broadband light source. Similarly, control of the narrowband light source by the controller may comprise activating or deactivating (switching on or off) the narrowband light source, and / or controlling a brightness of the narrowband light source.

[0016] The controller is further configured to capture image data with the image sensor. Thus, the controller may be connected to the image sensor in order to capture (acquire) images with the image sensor. The controller may comprise a memory for storing the captured image data. Additionally or alternatively, the controller may be configured to transmit the captured image data to an external device.

[0017] As the controller controls both the light sources and the image sensor, the controller can synchronise image capture with activation of the light sources, to capture images under different illumination conditions.

[0018] The oral probe device may further comprise an optical filter arranged to filter light incident on the image sensor. The optical filter may be configured to block light in a wavelength band emitted by the narrowband light source. In this manner, the image sensor does not receive the light emitted by narrowband light source. ThisP005261-W0015avoids light from the narrowband light source saturating the image sensor, so that relatively low intensity fluorescence can be detected.

[0019] Thus, the optical filter may be configured to block light in the wavelength band emitted by the narrowband light source, and to transmit fluorescence light stimulated in the mouth by the narrowband light source. For example, dentine fluorescence light may be green, whilst plaque fluorescence may be red. Accordingly, the optical filter may transmit green and red light, and block blue and / or ultraviolet light.

[0020] The optical filter may comprise a long pass filter having a cut-on wavelength above a central wavelength of the wavelength band emitted by the narrowband light source. In this manner, all wavelengths above the cut-on wavelength may be transmitted by the optical filter. This may ensure that visible light across a broad spectrum can be transmitted through the optical filter, to provide natural-looking images when the broadband light source is used.

[0021] The cut-on wavelength may be below a wavelength of the stimulated fluorescence, so that the stimulated fluorescence is transmitted through the filter. Thus, the cut-on wavelength may be between the wavelength of the narrowband light source and the wavelength of the stimulated fluorescence. The cut-on wavelength may be selected to be close to the wavelength of the narrowband light source, in order to maximise a proportion of the visible spectrum that is transmitted by the optical filter. Thus, the long pass filter enables effective fluorescence imaging and broadband imaging in the mouth.

[0022] Here, the cut-on wavelength of the long pass filter may refer to a wavelength above which the filter transmits light, and below which the filter blocks light.

[0023] Alternatively, other types of filter may be used. For example, the optical filter may comprise a band-stop (or band-rejection) filter, which is configured to block light in the wavelength band emitted by the narrowband light source.

[0024] The narrowband light source may have a central (or peak) wavelength in a range of 400 to 450 nm. Such a wavelength may cause features of interest such as plaque and / or dentine in the user’s mouth to fluoresce, thus facilitating detection and / or visualisation of the features of interest. As an example, the narrowband light source may have a central wavelength of 405 nm.P005261-W0016

[0025] The oral probe device may comprise a toothbrush. The toothbrush may comprise an electric toothbrush, e.g. comprising a motor configured to move the bristles.

[0026] The image sensor, the broadband light source, and the narrowband light source may be disposed in or next to a head of the toothbrush. In this manner, the image sensor is arranged to image an area of the user’s mouth that is next to (e.g. in front of) the toothbrush head. For example, a user could image an area they are about brush, or have just brushed, e.g. to check if the area needs cleaning.

[0027] The head of the toothbrush may correspond to a portion of the toothbrush comprising bristles for brushing teeth. The image sensor and the light sources may be arranged on a same side of the head as the bristles, e.g. so that the image sensor images an area in front of the bristles.

[0028] The controller may be configured to capture, with the image sensor, a pair of images comprising a first image captured in a first imaging mode and a second image captured in a second imaging mode. In the first imaging mode the broadband light source is activated and the narrowband light source is deactivated. In the second imaging mode the broadband light source is deactivated and the narrowband light source is activated. In this manner, the oral probe device can be used to automatically capture a pair of images where each of the light sources is individually activated. Accordingly, this may provide a first broadband (e.g. white light) image of the mouth, and a corresponding second fluorescence image (with the narrowband light source). In other words, a sequence of images which alternates between illumination with the broadband light source and the narrowband light source is obtained. The controller thus automatically synchronises activation of each of the light sources with operation of the image sensor, to capture the pair of images. In this manner, an area of the mouth may be inspected under two different illumination conditions. Capturing the pair of images may, for example, facilitate mapping fluorescent features in the second image onto the first image, e.g. to visualise the location of the fluorescent features in the broadband image. This may enable a user to readily identify areas in the mouth which may need further attention.

[0029] The pair of images may be captured in rapid succession. In this manner, the first image and the second image may be of (substantially) the same area in theP005261-W0017mouth. For example, the controller may be configured to obtain a video feed from the image sensor, the video feed comprising a plurality of images (frames) per second. The controller may then be configured to capture the pair of images such that the first image and the second image are successive images in the video feed.

[0030] The controller may be configured to alternate between the first imaging mode and the second imaging mode to capture a sequence of pairs of first and second images. In this manner, as the oral probe device is moved around the mouth, the controller can automatically capture pairs of images of different areas in the mouth. This may facilitate rapidly capturing pairs of narrowband and broadband images in the user’s mouth, without a user having to manually switch on an off the different light sources. This may in turn facilitate identifying features of interest in the mouth, using the combination of narrowband and broadband images.

[0031] Where, as mentioned above, the controller receives a video feed from the image sensor, the controller may be configured to alternate between the first imaging mode and the second imaging mode for each successive image (frame) in the video feed. In this manner, successive images (frames) in the video feed alternate between the first imaging mode (broadband light source activated) and the second imaging mode (narrowband light source activated). Thus, for example, where the video feed comprises a plurality of frames per second, one half of the plurality of frames will be captured in the first imaging mode and another half of the plurality of frames will be captured in the second imaging mode.

[0032] One or more image capture parameters of the image sensor may be different in the second imaging mode compared to the first imaging mode. For example, exposure time, gain, or some other image capture parameter may be adjusted between the first and second imaging modes. In this manner, operation of the image sensor may be adapted to the light sources used in the first and second imaging modes, respectively. This may enable a quality of the images captured in the two imaging modes to be improved, by optimising the image capture parameters for each light source.

[0033] In a second aspect, the invention provides an oral inspection system comprising: an oral probe device according to the first aspect; and a processing module configured to receive the image data from the image sensor, and to generateP005261-W0018a display signal based on the received image data. In this manner, the image data can captured with the image sensor can be displayed, so that a user can view images captured in the mouth. For example, the display signal may be provided to a display unit (e.g. screen) to provide a visualisation for the user. This may enable (substantially) real-time viewing of the image data and / or information derived from the image data, enable a user to inspect the mouth in real-time as they move the oral probe device around the mouth.

[0034] The display signal may be configured to cause a display unit (e.g. screen) to display information obtained from the image data. The displayed information may comprise, for example, a representation of the image data, and / or other information obtained by processing (analysing) the image data.

[0035] The system may further comprise a display unit configured to receive the display signal, and to display a representation of the image data. Additionally or alternatively, the system may comprise a memory configured to store the display signal. The display signal may then be accessed at a later time, e.g. to enable review of the image data at a later time.

[0036] The processing module may be in the oral probe device, such that all of the processing is performed locally at the oral probe device. For example, the processing module may be implemented by the controller in the oral probe device, e.g. as software or an application running on (executed by) the controller.

[0037] Alternatively, the processing module may be external to the oral probe device, e.g. the processing module may be at a host (or client) device. In such a case, the image data may be transmitted (e.g. streamed) to the client device, e.g. via a suitable wired or wireless connection. For example, the oral probe device may a communication module configured to transmit the image data to the host device. The host device may, for example, be a smartphone, tablet or laptop computer, or other suitable computing device.

[0038] Where the controller of the oral probe device is configured to capture a pair of images, as described above, the processing module may be configured to generate the display signal based on the first image and the second image. In this manner, the display signal may include information obtained from both the first image (broadband illumination) and the second image (narrowband illumination). This mayP005261-W0019enable features from both images to be visualised together, so that a user can rapidly identify features of interest. For example, the processing module may be configured to generate a composite image comprising information from the first image and the second image.

[0039] The processing module may be configured to detect a fluorescent feature in the second image. The processing module may further be configured to identify a location in the first image corresponding to the fluorescent feature. The display signal may comprise a representation of the first image with an indication of the identified location of the fluorescent feature. In this manner, a user can easily view of the location of the fluorescent feature in the first image which was captured with the broadband light source. As discussed above, the first image may have more naturallooking colours, such that different objects in the first image may be more clearly discernible and identifiable. Accordingly, by indicating the location of the fluorescent feature in the first image, this may provide a clear view of the location of the fluorescent feature with respect to other objects in the mouth.

[0040] As an example, the fluorescent feature in the second image may be plaque and / or dentine. As noted above, plaque may produce red fluorescence light and dentine may produce green fluorescence light, in response to a blue or ultraviolet narrowband light source. Accordingly, the processing module may be configured to detect the fluorescent feature based on a colour of the fluorescent feature. For example, the processing module may be configured to detect (identify) features in the second image having a colour within one or more predetermined colour ranges.

[0041] In some cases, the processing module may comprise a machine learning model that is configured to detect fluorescent features in the second image.

[0042] The location in the first image corresponding to the detected fluorescent feature may, for example, be determined by mapping a location of the fluorescent feature in the second image onto the first image. For example, the second image may be registered relative to the first image, to determine a location in the first image corresponding to the fluorescent feature. As the first image and the second image were captured with the same image sensor one after the other (e.g. in rapid succession), this facilitates registering the two images relative to one another.P005261-W00110

[0043] In some cases, the indication of the identified location of the fluorescent feature may comprise a mask at the identified location in the representation of the first image, the mask having a shape corresponding to the fluorescent feature. In other words, the processing module may generate a mask (e.g. segmentation mask) corresponding to the detected fluorescent feature in the second image. The mask may then be applied to the first image at the identified location. In this manner, the location and shape of the fluorescent feature (e.g. plaque, dentine) can be visualised in the broadband image.

[0044] The mask may be obtained by segmenting a region of the second image corresponding to the detected fluorescent feature.

[0045] The processing module may further be configured to determine a movement of the oral probe device between the first image and the second image, and to identify the location in the first image corresponding to the fluorescent feature based on the determined movement. This allows the processing module to compensate for any movement of the oral probe device between capture of the first and second images, so as to accurately identify the location of the fluorescent feature in the first image. Accordingly, if the user moves the oral probe device between capture of the first and second image, the processing module can take into account the determined movement to estimate the location of the fluorescent feature in the first image. This may improve an accuracy of the location at which the fluorescent feature is indicated in the first image.

[0046] The movement of the oral probe device between the first image and the second image may be determined in any suitable manner.

[0047] For example, the processing module may be configured to track (e.g.calculate) a displacement of one or more features in the second image relative to the first image, where the one or more features are visible in both the first and second images. Any suitable feature tracking algorithm or software may be used. The location of the fluorescent feature in the second image may then be shifted by an amount corresponding to the determined displacement, in order to determine the location in the first image corresponding to the fluorescent feature.

[0048] In some cases, the first image and the second image may be in a first pair of images captured with the image sensor; the processing module may be configured toP005261-W00111determine the movement of the oral probe device based on a comparison of the first image in the first pair of images with a first image in a second, subsequent pair of images captured with the image sensor. In other words, the controller can sequentially capture a first pair of images and a second pair of images with the image sensor (each pair having a first image and second image as described above). The movement of the oral probe device between the first and second images in the first pair is then determined based on a comparison of the first image in the first pair and the first image in the second pair, e.g. by interpolating between the two first images. For example, the processing module can track a displacement of one or more features visible in the first image of the first pair and the first image of the second pair.

[0049] It should be noted that compensation for displacement of the oral probe device may not always be needed. For instance, where the oral probe device is held sufficiently still and / or where the first and second images are captured sufficiently rapid succession, movement of the oral probe device may be negligible between capture of the first and second images, such that no compensation for movement of the oral probe device may be needed.

[0050] As mentioned above, the processing module may be configured to detect a fluorescent feature comprising plaque and / or dentine in the second image. Plaque may produce a red or orange fluorescence, whilst dentine may produce a green fluorescence. In this manner, both plaque and dentine can be detected, e.g. based on a colour of the fluorescence in the second image.

[0051] Where the oral probe device comprises an optical filter arranged to filter light incident on the image sensor the processing module may be configured to perform a colour correction process on the first image, and the display signal may be based on a colour-corrected version of the first image. In this manner, the colour correction enables compensation for the wavelengths removed by the filter, to provide a more natural-looking image. Where the display signal comprises a representation of the first image with an indication of the identified location of the fluorescent feature, the representation of the first image may be the colour-corrected version of the first image.

[0052] Various colour-correction techniques may be used. As an example, the colour-correction process may comprise boosting colours (wavelengths) in the firstP005261-W00112image corresponding to colours that are blocked by the optical filter. As another example, the colour-correction process may comprise adjusting colours in the first image based on a predetermined colour-mapping. The colour-correction process may further comprise performing a white balance process on the first image. In some cases, the colour-correction process may be performed with a machine learning model.

[0053] In a third aspect, the invention provides a method of inspecting a user’ s mouth with an oral probe device, the oral probe device comprising an image sensor, a broadband light source for illuminating a field of view of the image sensor, and a narrowband light source for stimulating fluorescence in the user’s mouth, the method comprising capturing, with the image sensor, a pair of images comprising a first image captured in a first imaging mode and a second image captured in a second imaging mode; wherein in the first imaging mode the broadband light source is activated and the narrowband light source is deactivated; and wherein in the second imaging mode the broadband light source is deactivated and the narrowband light source is activated.

[0054] The method of the third aspect may be performed with the oral probe device and / or oral inspection system of the preceding aspects. Accordingly, any features described in relation to the preceding aspects are equally applicable to the method of the third aspect.

[0055] The method may further comprise capturing a sequence of pairs of first images and second images by alternating between the first imaging mode and the second.

[0056] The method may further comprise generating a display signal based on the first image and the second image in the pair of images.

[0057] The method may further comprise detecting a fluorescent feature in the second image; and identifying a location in the first image corresponding to the fluorescent feature; and the display signal may comprise a representation of the first image with an indication of the identified location of the fluorescent feature.

[0058] The method may further comprise determining a movement of the oral probe device between the first image and the second image, and identifying the location inP005261-W00113the first image corresponding to the fluorescent feature based on the determined movement.

[0059] The first image and the second image may be in a first pair of images captured with the image sensor, and the method may further comprise determining the movement of the oral probe device based on a comparison of the first image in the first pair of images with a first image in a second, subsequent pair of images captured with the image sensor.

[0060] The method may comprise detecting a fluorescent feature comprising plaque and / or dentine in the second image.

[0061] The method may comprise performing a colour correction process on the first image, and wherein the display signal is based on a colour-corrected version of the first image.BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 shows a schematic diagram of an oral probe device;

[0063] Figure 2 shows an example transmission profile of an optical filter in the oral probe device, together with example emission profiles of light sources in the oral probe device;

[0064] Figure 3 shows an example of a sequence of images captured with the oral probe device.

[0065] Figure 4 shows a flow diagram of a method performed with the oral probe device.

[0066] Figure 5 shows examples of images obtained with the method of Figure 4;and

[0067] Figure 6 shows a flow diagram of another method performed with the oral probe device.DETAILED DESCRIPTION

[0068] Fig. 1 shows a schematic diagram of an oral probe device 10. The oral probe device 10 comprises an imaging module 12 including an image sensor 14, a broadband light source 16, and a narrowband light source 18. The imaging module 12 is located on a probe portion 20 of the oral probe device 10 which is insertableP005261-W00114into a user’s mouth for capturing images in the user’s mouth. The oral probe device 10 may further comprise a handle portion 22, connected to the probe portion 20. In some cases, the oral probe device 10 may be implemented as a toothbrush, e.g. an electric toothbrush. In such a case, the imaging module 12 may be arranged on or next to a brush head of the toothbrush.

[0069] The image sensor 14 is arranged to image an area of a user’s mouth. The image sensor 14 may, for example comprise camera such as a CCD or CMOS camera. The broadband light source 16 located next to the image sensor 14, to illuminate the area imaged by the image sensor 14. Generally speaking, the broadband light source 16 may emit light having a spectrum that covers a majority of the visible spectrum, to enable capture of images having natural-looking colours with the image sensor 14 when the broadband light source is activated. For example, the broadband light source 14 may comprise a white light LED.

[0070] The narrowband light source 18 is configured to stimulate fluorescence in features of interest in the user’s mouth such as plaque and / or dentine. Accordingly, the narrowband light source 18 may have a central (peak) wavelength in the range of 400 to 450 nm, e.g. 405 nm to stimulate fluorescence in the mouth. For example, the narrowband light source 18 may comprise a suitable narrowband LED, such as an ultraviolet LED or a blue LED. An example of a suitable narrowband light source is LHUV-0405-A070 from LUMILEDS HOLDING B.V., which has a central wavelength of 405 nm and a spectral half-width of 12 nm. The narrowband light source 18 is arranged next to the image sensor 14, such that fluorescence stimulated by the narrowband light source 18 can be imaged with the image sensor 14. In the example shown, the broadband light source 16 and the narrowband light source 18 are arranged on either side of the image sensor 14, however other arrangements are also contemplated. The imaging module 12 may further comprise a transparent cover and / or lens that covers the image sensor 14 and the light sources 16, 18, in order to protect them when the oral probe device 10 is inserted into the mouth.

[0071] The imaging module 12 further comprises an optical filter arranged to filter light incident on the image sensor 14. The optical filter is configured to block light in a wavelength band emitted by the narrowband light source 18, and to transmit fluorescence light (stimulated by the narrowband light source 18) to the image sensorP005261-W0011514. Fig. 2 shows an example transmission profile 25 of a long pass filter that can be used with the image sensor 14. The long pass filter has a cut-on wavelength Xc which is above the central wavelength k of an emission profile (wavelength band) 26 emitted by the narrowband light source 18, such that light emitted by the narrowband light source 18 is prevented from reaching the image sensor 14 by the filter. The cut- on wavelength Ze is below green fluorescence light 28 associated with dentine and orange / red fluorescence light 30 associated with plaque. For example, dentine may typically have a fluorescence wavelength around 490-500 nm with an excitation light of 405 nm, whilst plaque fluorescence may have a peak around 640 nm. The cut-on wavelength Ac may be selected so as to be relatively close to the wavelength Av of the narrowband light source 18, so as to minimise a proportion of the visible spectrum which is cut out by the optical filter, whilst still preventing transmission of light from the narrowband light source 18. As an example, Av may be around 405 nm, and Ac may be around 470 nm. Fig. 2 further shows an example emission profile 32 of the broadband light source 16. As can be seen, the broadband light source 16 has a much broader emission profile compared to the narrowband light source 18, such that images captured when the broadband light source 16 is activated will have a much more natural colour profile. In particular, the emission profile of the broadband light source 16 covers most or all of the visible spectrum, from about 400 nm to about 750 nm.

[0072] Returning to Fig. 1, the oral probe device 10 further includes an onboard controller 24, which is configured to control operation of the light sources 16, 18 and the image sensor 14, as described in more detail below. In particular, the controller 24 can switch each of the broadband light source 16 and the narrowband light source 18 on or off, and / or adjust a brightness (intensity) of each of the light sources 16, 18. For example, the controller 24 may be configured to control a respective switch for activating each of the light sources 16, 18, and / or an amount of power supplied to each of light source 16, 18. The controller 24 can further control the image sensor 14 to capture images with the image sensor 14. Although the controller 24 and wiring connecting the controller 24 to components of the imaging module 12 are shown in Fig. 1, in practice these will be located within a housing of the oral probe device 10P005261-W00116and so may not be visible. The housing may, for example, be a waterproof enclosure, e.g. made of plastic.

[0073] The controller 24 may be configured to capture images, with the image sensor 14, in a first imaging mode where the broadband light source 16 is activated and the narrowband light source 18 is deactivated. Images captured in the first imaging mode may be referred to as broadband images. The controller 24 may further be configured to capture images, with the image sensor 14, in a second imaging mode where the broadband light source 16 is deactivated and the narrowband light source 18 is activated. In some cases, the controller 24 may adjust one or more imaging parameters between the first imaging mode and the second imaging mode. For example, the controller 24 may adjust exposure time, gain, or some other parameter of the image sensor 14, to optimise image capture with the broadband and narrowband light sources, respectively. For example, a greater exposure time and / or gain may be used in the second imaging mode, as the narrowband light source 18 may be less bright than the broadband light source 16.

[0074] Fig. 3 shows an example sequence of images that can be capture by the controller 24 with the image sensor 14. With the oral probe device 10 arranged such that the imaging module 12 is in the mouth, a first image 34a is captured in the first imaging mode, i.e. with the broadband light source 16 activated and the narrowband light source 18 deactivated. Keeping the oral probe device 10 in the same position, a second image 36a in the second imaging mode, i.e. with the broadband light source 16 deactivated and the narrowband light source 18 activated. The first image 34a and the second image 36a thus constitute a first pair 38 of images of a same area of the mouth under different illumination conditions. The controller 24 can alternate over time between the first imaging mode and the second imaging mode, to capture a sequence of pairs of images, each pair having a first image and a second image in a similar fashion to the first pair 38. Thus, Fig. 3 shows a second pair 40 of images, including a first image 34b captured in the first imaging mode and a second image 36b captured in the second imaging mode. Of course, the order in which images are captured in the first and second imaging modes can be reversed.

[0075] In some cases, each image in the sequence captured by image sensor 14 may correspond to a respective frame in a video feed received from the image sensor 14.P005261-W00117Accordingly, the controller 245 may be configured to synchronise switching on and off of the light sources 16, 18 with a frame rate of the video feed, such that successive frames in the video alternate between illumination with the broadband light source 16 and the narrowband light source 18. As an example, a video feed obtained from the image sensor 14 may have a frame rate of N (e.g. 30) frames per second, with alternate (successive) frames being captured in the first imaging mode and the second imaging mode, respectively. Thus, N / 2 (e.g. 15) pairs of images may be captured per second, each pair including a first image captured in the first imaging mode, and a second image captured in the second imaging mode.

[0076] Fig. 4 shows a flow diagram of a method 40 of processing images captured with the oral probe device 10. The method 40 may be performed entirely with the controller 24, such that all of the processing steps may be performed locally at the oral probe device 10. Alternatively, some of the processing steps may be performed remotely from the oral probe device 10. For example, the oral probe device 10 may be configured to transmit images captured with the image sensor 14 to a separate host device. The host device may, for example, be a smartphone, tablet or laptop computer, or other suitable computing device. The onboard controller 24 may therefore comprise a communication module for wireless communication with the host device. For example, the oral probe device 10 may be configured to communicate with the host device via a Bluetooth® connection, via a Wi-Fi network, or other suitable wireless communication method. Additionally or alternatively, the oral probe device 10 may comprise a connector for providing a wired connection between the oral probe device 10 and the host device, to enable wired communication.

[0077] In a step 42 of the method 40, a pair of images is captured with the image sensor 14, the pair of images comprising a first image captured in the first imaging mode and a second image captured in the second imaging mode. For example, the first pair 38 of images 34a, 36a described above may be captured in step 42. Step 42 may be performed by the controller 24.

[0078] In a step 44 of the method 40, a colour-correction process is performed on the first image in the pair captured in step 42. The colour-correction process is intended to compensate for wavelengths that are blocked by the optical filter, to provide aP005261-W00118more natural-looking broadband (e.g. white light) image. Various different colour correction techniques may be used. As an example, the colour-correction process may comprise boosting colours in the first image corresponding to colours that are blocked by the optical filter. For instance, blue colours may be boosted in the first image, i.e. intensity levels for blue pixels may be increased. As another example, a white balancing process may be performed on the first image, to improve a colour balance in the first image.

[0079] As a further example, a colour-mapping process may be performed. In such a colour-mapping process, a predetermined look-up table may be provided, which maps (i.e. associates) a set of input colour values (e.g. RGB values) with a corresponding set of output colour values. Then, for each pixel in the first image, the colour value of the pixel is updated based on the look-up table (e.g. by determining the output colour value associated with the original colour value of the pixel). In this manner, a colour-corrected version of the first image can be produced. The predetermined table can be obtained via suitable testing, e.g. by capturing broadband images with the oral probe device 10 and determining adjustments to the colour values in the images to provide more natural-looking images.

[0080] As yet a further example, the colour-mapping process may be performed using a machine learning model. The machine learning model may be configured to receive the first image as an input, and to generate a colour-corrected version of the first image as an output. An example of a suitable model includes the HistoGAN model, which is a generative adversarial network with an application in recolouring of images based on input reference images, through the use of colour histograms (see e.g. https: / / arxiv.org / abs / 2011.11731). The machine learning model may be trained using a supervised learning process with a training dataset comprising a plurality of images of one or more persons’ mouths captured with the oral probe device 10 (and / or with a camera having a similar optical filter). The training dataset may further comprise a set of ground truth data, the ground truth data comprising a colour- corrected version of each of the images in the training dataset. In this manner, the machine learning model can learn to correct colours in images captured by the oral probe device 10. The colour-corrected versions in the ground truth data may be obtained in any suitable manner, for example by performing a colour-mappingP005261-W00119process as described above. Alternatively, a self-supervised approach may be used for image recolouring. For example, the paper “Neural Image Recolorization for Creative Domains” (Boyi Li et al., IEEE / CVPRW 2022) describes a self-supervised recolorization approach that is applicable to the images captured with the image sensor 14.

[0081] In a step 46 of the method 40, a detection process is performed on the second image to detect fluorescent features of interest in the second image. For example, the detection process may be configured to detect fluorescence arising from plaque in the user’s mouth. As noted above, fluorescence arising from plaque may be in a predetermined wavelength range, such that it is visible as orange or red light. In contrast, dentine in the teeth may produce green fluorescence, such that areas on the teeth having plaque may be readily distinguishable based on the orange / red colour of the fluorescence. Accordingly, in step 46 the second image may be analysed to identify areas in the second image having orange or red intensity, which may be indicative of plaque fluorescence. For example, the detection process may comprise identifying areas in the second image having colour values (e.g. RGB values) falling in a predetermined range (e.g. corresponding to orange and / or red fluorescence) and having an intensity above a predetermined threshold (e.g. to avoid picking up background noise).

[0082] In some cases, a machine learning model may be used in step 46 for detecting fluorescent features in the second image. The machine learning model may be configured to perform object detection on the second image, to detect fluorescent features of interest, e.g. plaque. By way of example, the machine learning model may comprise the SSD MobileNetV2 model, which is a supervised deep learning model that utilises a Single Shot MultiBox Detector (SSD) layer on top of the MobileNetV2 architecture ( seetf2 detect! on zoo . n ■ d ) . The machine learning model may further be configured to segment areas of the image corresponding to detected features of interest. The machine learning model may be trained using a supervised learning process with a training dataset comprising a plurality of images of one or more persons’ mouths captured with the oral probe device 10 (and / or with a camera having a narrowbandP005261-W00120light source). The images in the training dataset may be labelled to indicate areas in the images corresponding to fluorescent features of interest. In this manner, the machine learning model can learn to detect and segment fluorescent features of interest in images captured by the oral probe device 10.

[0083] Following step 46, the method comprises a step 48 in which a mask corresponding to the detected fluorescent feature(s) is generated. The mask may therefore have a shape and coordinates corresponding to the detected fluorescent feature(s) in the second image, e.g. such that when the mask is applied to the second image the mask covers the detected fluorescent feature(s). The mask may thus be a segmentation mask which identifies the detected fluorescent feature(s). For example, where the image is analysed to determine areas of red or orange intensity as described above, the mask may be generated so as to cover detected areas of red or orange intensity. Where a machine learning model is used to detect the fluorescent features, the machine learning model may be configured to output a segmentation mask corresponding to detected fluorescent features.

[0084] At step 50, the method 40 comprises applying the mask generated in step 48 to the colour-corrected first image obtained from step 44. In this manner, the location and shape of the fluorescent feature detected in the second image, is indicated in the first image. In particular, the coordinates of the mask in the second image are used to map the mask onto the first image, e.g. by applying the mask at the same coordinates in the first image as in the second image. For example, where the oral probe device 10 is held relatively still between captured of the first image and the second image, or where the first and second image are captured in sufficiently quick succession, the location of the fluorescent features may be the same in both images.

[0085] Fig. 5 shows an example of a pair of images comprising a first image 54 and a second image 56 captured with the oral probe device 10, as described above. The second image 56, captured in the second imaging mode, includes a mask 58 indicating a location and shape of a detected fluorescent feature, as described above in relation to step 48. The mask 58 is applied to same location in the first image 50, as described in relation to step 50.

[0086] Following step 50, the method 40 comprises a step 52 of generating a display signal, for displaying the combined image obtained from step 50. For example, theP005261-W00121display signal may be provided to a display unit (e.g. screen), to cause the display unit to display the first image with the mask, as obtained from step 50. Additionally or alternatively, the display signal may be stored in a memory, e.g. for later viewing and / or analysis.

[0087] Step 44 may be performed in parallel (e.g. simultaneously) with steps 46 and 48. In some cases, the colour correction step 44 may be omitted, e.g. such that the mask is applied in step 50 to the original first image.

[0088] In some implementations, all of steps 44-52 may be performed by the controller 24, e.g. with processing module in the form of software installed on the controller 24. In this manner, all of the image analysis may be performed locally at the oral probe device 10. Alternatively, in line with the above discussion, the processing module may be implemented in a separate host device, such that processing of the images is performed remotely from the oral probe device 10. In such a case, following capture of the images in step 42, the images may be transmitted from the oral probe device to the processing module in the host device, so that steps 44-52 can be performed at the host device.

[0089] Fig. 6 illustrates another method 60 of processing images captured with the oral probe device 10. The method 60 operates in a similar manner to the method 40 described above, but further compensates for possible movement of the oral probe device 10 between capture of the first and second images.

[0090] Fig. 6 shows a sequence of images captured with the oral probe device 10, including first pair 66 of images consisting of first and second images 62a, 64a, and a second pair 72 of images consisting of first and second images 68b, 70b. The first pair 66 and second pair 72 of images may be captured one after the other with the image sensor 14, as described above in relation to Fig. 3.

[0091] In a step 74, the method 60 comprises performing a colour-correction process on the first image 62a of the first pair 66. Likewise, in step 76 the method 60 comprises performing a colour-correction process on the first image 68b of the second pair 72. The colour-correction process in steps 74 and 76 may be performed as described above in relation to step 44 of the method 40.

[0092] In a step 78, the method 60 comprises detecting a fluorescent feature in the second image 64a of the first pair 66, and generating a mask corresponding to theP005261-W00122detected fluorescent feature. The step 78 may be performed as described in relation to steps 46 and 48 of the method 40 described above.

[0093] In a step 80, the method 60 comprises determining a movement of the oral probe device 10 between capture of the first image 62a and the second image 64a of the first pair 66. This may be achieved by comparing the first image 62a in the first pair 66 with the first image 68b in the second pair 72, to track motion of the oral probe device between capture of the two images 62a, 68b. For example, locations of one or more features visible in the first image 62a in the first pair 66 and the first image 68b in the second pair 72 may be compared, to determine a displacement of the oral probe device 10 between capture of the two images 62a, 68b. In particular, object detection can be performed on each of the first images 62a, 68b to detect one or more objects in common between the two images. The locations of the one or more objects in the two images 62a, 68b may then be compared to determine the displacement of the oral probe device 10. By way of example, known computer vision techniques such as optical flow estimation using the Lucas-Kanade method may be used for tracking movement of the oral probe device 10. Such a method is implemented in a number of available software packages such as OpenCV. Other computer vision techniques for motion tracking may also be used.

[0094] Once the displacement of the oral probe device 10 between the first image 62a in the first pair 66 and the first image 68b in the second pair is determined, an interpolation may be performed to determine the displacement of the oral probe device 10 between the first and second images 62a, 64a in the first pair 66. For example, where the image sensor 14 is configured to capture images at regular intervals (e.g. at a predetermined frame rate), a time of capture of the second image 64a in the first pair 66 may be half-way between a time of capture of the first image 62a in the first pair 66 and a time of capture of the first image 68 in the second pair 72. Accordingly, it can be assumed that a displacement of the oral probe device 10 between the first and second images 62a, 64a in the first pair 66 will be approximately half of the displacement between capture of the first images 62a, 68b of the first and second pairs. Accordingly, the displacement of the oral probe device 10 between capture of the first and second images 62a, 64a of the first pair 66 may be determinedP005261-W00123as half of the displacement between the first image 62a in the first pair 66 and the first image 68b (as obtained from the motion tracking process described above).

[0095] By way of example, the motion tracking process described above may yield a displacement of (x, y) of the oral probe device 10 in the first image 68b of the second pair 72 with respect to the first image 62a of the first pair. Here, x and y represent values of the displacement along an x-axis (e.g. horizontal axis) and a y- axis (e.g. vertical axis) of the first image 62a, respectively. The displacement of the oral probe device 10 between capture of the first and second images 62a, 64a in the first pair 66 may then be determined to be half the displacement between the first images 62a, 68b, i.e. (x / 2, y / 2).

[0096] In step 82, the method 60 comprises translating the mask obtained in step 78, in order to compensate for the movement of the oral probe device 10 determined in step 80. In particular, the position of the mask is shifted to identify the location of the detected fluorescent feature in the first image 62a. Thus, the coordinates of the mask in the second image are shifted by an amount corresponding to the determined displacement, to identify corresponding coordinates in the first image 62a. For instance, with the example above where the determined displacement is (x / 2, y / 2), the coordinates of the mask in the second image 64a may be shifted by (-x / 2, -y / 2) to determine coordinates for the mask in the first image 62a.

[0097] At step 84, the method 60 comprises applying the mask with translated (shifted) coordinates obtained in step 82, to the colour-corrected version of the first image 62a of the first pair 66 obtained in step 74. Then, in step 86, a display signal is generated for displaying the image with the mask obtained from step 84, in a manner analogous to the step 52 described above.

[0098] In the described implementation of the method 60, the mask is applied to the first image 62a of the first pair 66 in step 84. However, in other implementations, the mask may instead be applied to the first image 68b of the second pair 72. In such a case, step 82 may be modified to determine coordinates for the mask in the first image 68b of the second pair 72, using the displacement determined in step 80. For instance, using the example above, the coordinates of the mask in the second image 64a may be shifted by (x / 2, y / 2) to determine coordinates for the mask in the first image 68b.P005261-W00124

[0099] In line with the above discussion, steps 74-86 may be performed locally by a processing module implemented with the controller 24 of the oral probe device 10. Alternatively, the steps 74-86 may be performed remotely, e.g. with a processing module implemented in a separate host device that receives the captured images from the oral probe device 10. In some implementations, the colour correction steps 74, 76 may be omitted, such that the method 60 is performed with the original broadband images 62a, 68b.

[0100] The examples described above are illustrative of the present disclosure, and further examples are envisaged. It is to be understood that any feature described in relation to any one example may be used alone or in combination with other features of the example, and may also be used in combination with one or more features of any other of the examples, or any combination of any other of the examples. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the disclosure, which is defined in the accompanying claims.

Claims

1. P005261-W00125CLAIMS1. An oral probe device for insertion into a user’s mouth, the oral probe device comprising:an image sensor;a broadband light source for illuminating a field of view of the image sensor; a narrowband light source for stimulating fluorescence in the user’s mouth; anda controller configured to control operation of the broadband light source and the narrowband light source, and to capture image data with the image sensor.

2. An oral probe device according to claim 1, further comprising an optical filter arranged to filter light incident on the image sensor, wherein the optical filter is configured to block light in a wavelength band emitted by the narrowband light source.

3. An oral probe device according to claim 2, wherein the optical filter comprises a long pass filter having a cut-on wavelength above a central wavelength of the wavelength band emitted by the narrowband light source.

4. An oral probe device according to any preceding claim, wherein the narrowband light source has a central wavelength in a range of 400 to 450 nm.

5. An oral probe device according to any preceding claim, wherein the oral probe device comprises a toothbrush.

6. An oral probe device according to any preceding claim, wherein the controller is configured to capture, with the image sensor, a pair of images comprising a first image captured in a first imaging mode and a second image captured in a second imaging mode;wherein in the first imaging mode the broadband light source is activated and the narrowband light source is deactivated; andP005261-W00126wherein in the second imaging mode the broadband light source is deactivated and the narrowband light source is activated.

7. An oral probe device according to claim 6, wherein the controller is configured to alternate between the first imaging mode and the second imaging mode to capture a sequence of pairs of first and second images.

8. An oral probe device according to claim 6 or 7, wherein one or more image capture parameters of the image sensor are different in the second imaging mode compared to the first imaging mode.

9. An oral inspection system comprising:an oral probe device according to any preceding claim; anda processing module configured to receive the image data from the image sensor, and to generate a display signal based on the received image data.

10. An oral inspection system according to claim 9, wherein the oral probe device is according to one of claims 6 to 8, and wherein the processing module is configured to generate the display signal based on the first image and the second image.

11. An oral inspection system according to claim 10, wherein the processing module is configured to:detect a fluorescent feature in the second image;identify a location in the first image corresponding to the fluorescent feature; wherein the display signal comprises a representation of the first image with an indication of the identified location of the fluorescent feature.

12. An oral inspection system according to claim 11, wherein the indication of the identified location of the fluorescent feature comprises a mask at the identified location in the representation of the first image, the mask having a shape corresponding to the fluorescent feature.P005261-W0012713. An oral inspection system according to claim 11 or 12, wherein the processing module is further configured to determine a movement of the oral probe device between the first image and the second image, and to identify the location in the first image corresponding to the fluorescent feature based on the determined movement.

14. An oral inspection system according to claim 13, wherein:the first image and the second image are in a first pair of images captured with the image sensor;the processing module is configured to determine the movement of the oral probe device based on a comparison of the first image in the first pair of images with a first image in a second, subsequent pair of images captured with the image sensor.

15. An oral inspection system according to any of claims 11 to 14, wherein the processing module is configured to detect a fluorescent feature comprising plaque and / or dentine in the second image.

16. An oral inspection system according to any of claims 9 to 15, wherein the oral probe device is according to claim 2, wherein the processing module is configured to perform a colour correction process on the first image, and wherein the display signal is based on a colour-corrected version of the first image.

17. A method of inspecting a user’s mouth with an oral probe device, the oral probe device comprising an image sensor, a broadband light source for illuminating a field of view of the image sensor, and a narrowband light source for stimulating fluorescence in the user’s mouth, the method comprising capturing, with the image sensor, a pair of images comprising a first image captured in a first imaging mode and a second image captured in a second imaging mode;wherein in the first imaging mode the broadband light source is activated and the narrowband light source is deactivated; andwherein in the second imaging mode the broadband light source is deactivated and the narrowband light source is activated.P005261-W0012818. A method according to claim 17, further comprising capturing a sequence of pairs of first images and second images by alternating between the first imaging mode and the second.

19. A method according to claim 17 or 18, further comprising generating a display signal based on the first image and the second image in the pair of images.

20. A method according to claim 19, further comprising:detecting a fluorescent feature in the second image; andidentifying a location in the first image corresponding to the fluorescent feature;wherein the display signal comprises a representation of the first image with an indication of the identified location of the fluorescent feature.

Images