Oral inspection system

Abstract

An oral inspection system is described. The oral inspection system comprises an oral probe device (10) for insertion into a user's mouth, the oral probe device comprising an image sensor (18), and a light source (20) for illuminating an area imaged by the image sensor. The oral inspection system further comprises a controller (24) configured to determine if the oral probe device is in the user's mouth; and control a brightness of the light source as a function of the determination.

Description

P005258-W0011ORAL INSPECTION SYSTEMBACKGROUND

[0001] Good oral hygiene relies on regular cleaning inside the mouth, including brushing the teeth and caring for the gums and tongue. Such cleaning activities can be complemented by inspection of the mouth, to ensure that cleaning is being performed adequately and identify areas in the mouth which may require further attention. Inspection of the mouth may be performed using various techniques, such as visual inspection by a dentist, and / or using an imaging device that is configured to capture images inside the mouth. Such an imaging device may, for example, comprise a camera which is insertable into the user’s mouth for capturing images of the teeth and gums. The images may then be displayed on a screen, to be reviewed by the user or dentist.SUMMARY

[0002] The invention provides an oral inspection system comprising: an oral probe device for insertion into a user’s mouth, the oral probe device comprising an image sensor, and a light source for illuminating an area imaged by the image sensor; and a controller configured to: determine if the oral probe device is in the user’s mouth; and control a brightness of the light source as a function of the determination.

[0003] In this manner, the brightness (e.g. intensity) of the light source is automatically controlled based on whether the oral probe device is located in the user’s mouth or not. For example, the brightness could be increased when the oral probe device is located in the user’s mouth, or the light source could be controlled so that it is only activated when the oral probe device is in the user’s mouth. This may facilitate capturing images inside the user’s mouth with the image sensor, as the brightness of the light source may be automatically adjusted to provide suitable illumination conditions in the user’s mouth. Accordingly, the user need not manually control (e.g. switch on and off) the light source, thus greatly facilitating the imaging procedure. Moreover, this may avoid accidentally shining the light source with high brightness into the user’s eyes, which in some cases could dazzle or cause discomfort to the user. For instance, the light source could be switched off, or the brightness ofP005258-W0012the light source could be reduced, when the oral probe device is outside the user’s mouth. This contributes to improving safety and user-friendliness of the oral inspection system. Automatic control of the light source also contributes to improving a power efficiency of the oral inspection system, e.g. by only activating the light source or increasing the brightness when the oral probe device is in the mouth.

[0004] The oral probe device may comprise a handle (or handpiece), and a probe portion which is insertable into the user’s mouth. Thus, in use, the probe portion may be inserted into the mouth, whilst the handle may remain outside the mouth. The probe portion may, for example, comprise an elongate shaft that extends from the handle. The image sensor and the light source may be coupled to the probe portion, to enable illumination and imaging of the mouth when the probe portion is inserted into the mouth. For example, the image sensor and / or the light source may be disposed (mounted) on the probe portion, e.g. near a distal end of the probe portion.

[0005] Alternatively, the image sensor and the light source may be located in the handle, with a respective light guide arranged to optically couple the image sensor and light source to a distal end of the oral probe device. For example, a fibre optic light guide or a light pipe may be used for this purpose. The light guide would transmit light from the light source at the handle to the distal end of the probe, where it can illuminate the mouth. Similarly, the light guide would receive reflected light from the mouth and direct the light to the image sensor. The light guide could be made of fiber optics or transparent plastic, designed to bend or curve along a length of the oral probe device to deliver and collect light at the probe's tip. Positioning the image sensor and the light source in the handle may allow for more flexibility in the design of the probe portion, e.g. enabling a reduction in size which may improve ergonomics, while still providing effective illumination and imaging of the mouth.

[0006] The image sensor may comprise any suitable image sensor, such as a camera (digital camera). The image sensor may comprise an array of photosensitive elements (e.g. pixels). For example, the image sensor may comprise a charge-coupled device (CCD) image sensor, or a complementary metal-oxide-semiconductor (CMOS) image sensor.P005258-W0013

[0007] The light source may, for example, comprise any suitable type of light source, such as light-emitting diode (LED). In some cases, the oral probe device may comprise multiple (e.g. two or more) light sources, each arranged to illuminate the area imaged by the image sensor.

[0008] The light source is arranged to illuminate the area imaged by the image sensor. In other words, a field of view of the image sensor is illuminated by the light source. Thus, objects imaged by the image sensor may be illuminated with the light source. For example, the light source may be located next to the image sensor, such that light emitted by the light source is directed into the area imaged by the image sensor.

[0009] The controller may comprise any suitable processing circuitry, device or system that is configured to perform the control operations described herein. In some cases, the controller may be disposed in the oral probe device. In other cases, the controller may be remote from the oral probe device, in which case the controller may be communicatively coupled to the oral probe device (e.g. via a communication link) to exchange data with the oral probe device.

[0010] Various techniques may be used by the controller for determining if the oral probe device is in the user’s mouth, examples of which are given below. Generally speaking, the controller determines if the oral probe device is in the user’s mouth based on information obtained from one or more sensors in the oral probe device.

[0011] Here, determining if the oral probe device is in the user’s mouth may comprise determining if a portion of the oral probe device including the light source and the image sensor is in the user’ s mouth. In other words, the controller determines if the oral probe device is arranged to capture an image inside the user’s mouth. For example, the controller may be configured to determine if the probe portion of the oral probe device is in the user’s mouth. Thus, the brightness of the light source is controlled based on whether the light source is in the user’s mouth or not.

[0012] Controlling the brightness of the light source may comprise, for example, increasing or decreasing the brightness of the light source, and / or activating or deactivating (switching on or off) the light source. For example, the controller may be configured to control power supplied to the light source to control the brightnessP005258-W0014of the light source. Here, the brightness of the light source may refer to an intensity of light emitted by the light source.

[0013] The controller may be configured to receive image data from the image sensor, and to determine if the oral probe device is in the user’s mouth using the image data. Using images captured by the image sensor may enable accurate detection of when the oral probe is in the user’s mouth, in turn allowing highly accurate and responsive automatic control of the light source in response to the position of the oral probe device with respect to the user’s mouth. In this manner, the image sensor may serve the dual functions of imaging the user’ s mouth, and detecting when the oral probe device is in the user’s mouth. This may avoid having to use a separate sensor for detecting when the oral probe device is inserted into the user’s mouth, thus simplifying construction of the device.

[0014] The image data received by the controller may comprise one or more images captured by the image sensor. The controller may be configured to analyse the image data, to determine if the image data was captured inside or outside the user’s mouth. The controller may comprise an image analysis algorithm for analysing the image data to perform the determination. In other words, the image analysis algorithm receives the image data as an input, and in response generates as an output a determination of whether the oral probe device is in the user’s mouth or not. For example, the image analysis algorithm may be configured to classify the image data as being captured inside the mouth or outside the mouth.

[0015] Various image analysis techniques may be used by the controller for this purpose, some of which may involve machine learning techniques and / or traditional image processing techniques. In general, the controller may be configured to detect one or more visual features in the image data (examples of which are given below), to determine if the oral probe device is in the user’s mouth.

[0016] The controller may comprise a machine learning model that is configured to determine if the oral probe device is in the user’s mouth using the image data. The machine learning algorithm may, for example, comprise an image classifier that is configured (trained) to classify the received image data as corresponding to inside the mouth or outside the mouth. Such a machine learning model may comprise, for example, an artificial neural network such as a convolutional neural network.P005258-W0015

[0017] The controller may be configured to perform a colour analysis of the image data to determine if the oral probe device is in the user’s mouth. Images captured inside the mouth will predominantly include certain colours and / or colour patterns, e.g. arising from the teeth, gums, tongue, etc., such that the colour distribution of an image can be analysed to determine if the image was captured inside the mouth or not. Accordingly, analysing the colours of the captured images can provide accurate classification of images to determine if the oral probe device is in the mouth or not.

[0018] By way of example, a statistical analysis of RGB values or alternative colour spaces like HSV and Lab in the captured images can help differentiate between teeth, gums, and background, e.g. based on mean, variance, and histogram distributions. Spatial relationships between colours, such as pink (gums) next to white (teeth), can be analysed using colour segmentation or co-occurrence matrices. Machine learning models, including traditional classifiers (e.g., Random Forest or SVM) using engineered colour features, or deep learning-based convolutional neural networks (CNNs) trained on labelled images, can learn complex colour and texture patterns. Additionally, threshold-based colour segmentation in HSV space can help isolate relevant regions, allowing for rule-based classification. Combining these techniques can improve accuracy in distinguishing in-mouth versus out-of-mouth scenarios.

[0019] The controller may be configured to perform a texture analysis of the image data to determine if the oral probe device is in the user’s mouth. Structures inside the mouth, such as teeth, gums, tongue, etc., have different surface textures which can be detected through analysis of images captured in the mouth. Accordingly, the captured images can be analysed to determine the surface texture of imaged objects. Then, if certain predetermined surface textures, and / or combinations of surface textures, are detected, the controller may determine that the images were captured inside the mouth. For example, Local Binary Patterns (LBP) or Gabor Filters are known techniques for extracting and / or analysing surface textures in computer vision. By way of example, suitable texture detection and classification techniques are provided in the article “Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns”, Ojala, T., Pietikainen, M., & Maenpaa, T. (2002), IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7), 971-987.P005258-W0016

[0020] Additionally or alternatively, machine learning (deep-learning) techniques may be used for surface texture analysis and classification, such as a deep texture encoding network (e.g. as described in “Texture Encoding Network for Texture Recognition”, Zhang, EL, Xue, J., & Dana, K. (2017), CVPR), and / or a vision transformer (e.g. as described in “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2020)).

[0021] Where a machine learning model is used, the model could be configured (trained) to recognise (e.g. classify) predetermined surface textures corresponding to one or more objects in the mouth (e.g. teeth, gums, tongue, lip, cheek, etc.). The machine learning model may then be configured to determine that the image data was captured in the mouth if surface textures corresponding to one or more (in some cases two or more) of the predetermined surface textures are detected. The machine learning model may be trained, for example, using a supervised training process with a training dataset comprising a plurality of images of a mouth, where areas of the images are labelled to indicated one or more objects in the mouth (e.g. teeth, gums, tongue, etc.).

[0022] The controller may be configured to perform shape detection on the image data to determine if the oral probe device is in the user’s mouth. Structures inside the mouth, such as teeth, gums, tongue, etc., have certain general shapes such that shape detection can be used to determine if a captured image was captured inside the mouth. For example, the controller could be determined to detect shapes corresponding to teeth and / or gums to determine if the image data was captured in the mouth or not. For instance, a machine learning algorithm may be trained to detect (e.g. classify) shapes corresponding to objects in the mouth (e.g. teeth, gums, tongue, lip, cheek, etc.) in captured images.

[0023] For example, techniques such as Canny edge detection, contour detection, and Hough Transform can identify the outlines of teeth, gums, and other oral structures. Morphological operations and / or the Watershed algorithm can be used to help refine shape segmentation, especially when structures overlap. For more robust detection, machine learning models such as CNNs (e.g., ResNet, MobileNet) can classify oral structures, while segmentation models like U-Net and Mask R-CNN canP005258-W0017accurately delineate teeth, tongue, and gums. A hybrid approach combining traditional contour-based methods with deep learning segmentation can enhance accuracy in detecting whether an image was captured inside the mouth.

[0024] The oral probe device may further comprise a motion sensor for detecting a motion of the oral probe device, and the controller may be configured receive an output from the motion sensor and determine if the oral probe device is in the user’s mouth using the output from the motion sensor. It has been observed that the oral probe device generally undergoes different types of motion depending on whether it is inside or outside the user’s mouth. Accordingly, the motion sensor enables the controller to reliably determine whether the oral probe device is in the user’s mouth or not.

[0025] In some cases, the controller may use both the image data and the output from the motion sensor to determine if the oral probe device is in the user’s mouth. Combination of data from the image sensor and motion sensor may provide a more reliable and accurate determination result.

[0026] The motion detected by the motion sensor may comprise, for example, a change in position, an acceleration, and / or a change in angle (or orientation) of the oral probe device. Thus, the motion sensor may comprise various types of sensor (e.g. accelerometer, gyroscope), depending on the type(s) of motion to be detected.

[0027] The motion sensor may detect different motion patterns of the oral probe device, depending on whether the oral probe device is located in the user’s mouth or not. For example, a range of motion of the oral probe device may be much smaller (more restricted) in the user’s mouth, whereas larger motions may be detected when the oral probe device is outside the user’s mouth (e.g. when the device is picked up and moved towards the mouth). Accordingly, the controller may be configured to analyse the output from the motion sensor to determine whether a motion pattern of the oral probe device corresponds to the oral probe device being inside or outside the mouth.

[0028] Various techniques may be used by the controller for analysing the output from the motion sensor to determine if the oral probe device is in the user’s mouth or not. Example techniques may involve machine learning techniques and / or traditional data processing techniques. In general, the controller may be configuredP005258-W0018to detect one or more features in the output from the motion sensor (examples of which are given below), to determine if the oral probe device is in the user’s mouth.

[0029] As an example, the controller may be configured to determine that the oral probe device is in the user’s mouth if the output from the motion sensor satisfies a predetermined condition.

[0030] The motion sensor may be located in any suitable location in the oral probe device. For example, the motion sensor may be located in the handle (or main body) of the oral probe device.

[0031] The controller may be configured to determine if the oral probe device is in the user’s mouth based on (as a function of) one or more characteristics of the detected motion (or movement pattern) of the oral probe device. For example, the one or more characteristics may include an amplitude and / or a frequency of the detected motion. The oral probe device may typically undergo different movement patterns when it is in the mouth compared to when it is outside the mouth, such that the controller can determine whether the oral probe device is in the mouth or not based on characteristics such as movement amplitude and frequency. For example, smaller amplitude movements are more likely when the oral probe device is in the mouth, whereas larger amplitude movements are more likely when the oral probe device is outside the mouth. Similarly, the oral probe device may also be more likely to undergo higher frequency movements in the mouth, e.g. due to rapid back-and forth brushing motions where oral probe device comprises a toothbrush.

[0032] Thus, in cases where the oral probe device comprises a toothbrush, the controller may be configured to determine, from the output of the motion sensor, if the oral probe device is undergoing a motion corresponding to a toothbrushing movement pattern. If a toothbrushing movement pattern is detected, the controller can determine that the oral probe device is in the user’s mouth. For instance, the controller may store one or more predetermined toothbrushing movement patterns, to which the output from the motion sensor is compared. Additionally or alternatively, a machine learning model may be trained to recognise toothbrushing movement patterns from the output of the motion sensor.

[0033] In some cases, the motion sensor may be configured to detect a vibrational motion of the oral probe device, and the controller may be configured to determineP005258-W0019if the oral probe device is in the user’s mouth based on an amplitude of the vibrational motion. For example, the motion sensor may comprise an accelerometer for detecting the vibrational motion. The vibrational motion may result from a motor in the oral probe device, e.g. where the oral probe device is an electric toothbrush. When the oral probe device is inserted into the user’s mouth the amplitude of the vibrations may be dampened due to contact with the user’s mouth, e.g. compared to when the brush is outside the mouth. For instance, where the oral probe device comprises a toothbrush, the vibrations may be dampened by when the toothbrush is placed against the teeth (e.g. during brushing). Accordingly, a reduction in amplitude of the vibrational motion may be indicative of the oral probe device being inserted into the mouth.

[0034] Thus, the controller may be configured to determine that the oral probe device is in the user’s mouth if an amplitude of the detected vibrational motion is below a predetermined threshold.

[0035] The motion sensor may be configured to detect an orientation of the oral probe device, and the controller may be configured to determine if the oral probe device is in the user’s mouth based on the detected orientation. For example, the motion sensor may comprise a gyroscope for detecting the orientation (and / or a change in orientation) of the oral probe device. The output of the gyroscope may be indicative of the orientation (e.g. angle) of the oral probe device, e.g. in a frame of reference. It has been observed that the oral probe device may predominantly be used in certain orientations when it is inserted into the mouth. Accordingly, by detecting the orientation of the oral probe device, the controller can predict if the oral probe device is in the user’s mouth or not.

[0036] The controller may be configured to determine that the oral probe device is in the user’s mouth if the orientation of the oral probe device satisfies a predetermined condition. For instance, the controller may be configured to determine that the oral probe device is in the user’s mouth if the orientation of the oral probe device is within a predetermined range.

[0037] The motion sensor may be configured to output angular data indicative of at least two of a pitch, roll, and yaw of the oral probe device, and the controller may be configured to determine if the oral probe device is in the user’s mouth as a functionP005258-W00110of the angular data for each of the at least two of the pitch, roll, and yaw. In this manner, the controller uses angular data (e.g. values) corresponding to two or more of the pitch, roll, and yaw to determine if the oral probe device is in the user’s mouth. It has been found that this combination of angular data yields accurate determination results, as certain combinations of pitch, roll, and / or yaw angles may be consistent when the oral probe device is in the user’s mouth. The motion sensor may comprise a gyroscope, from which the angular data can be obtained. In particular, a gyroscope may output a signal indicative of angular velocity, from which angular data including pitch, roll, and yaw can be obtained using known mathematical models (often referred to as attitude estimators).

[0038] The pitch, roll, and yaw of the oral probe device may be defined with respect to a longitudinal axis of the oral probe device in a frame of reference.

[0039] Where the motion sensor is configured to output the angular data indicative of at least two of a pitch, roll, and yaw, the controller may further be configured to determine a region (or zone) of the mouth imaged by the image sensor when the oral probe device is in the mouth. Indeed, the oral probe device will generally need to be held in specific orientations to image different regions of the mouth, such as front upper teeth, front lower teeth, left upper teeth, left lower teeth, right upper teeth, right lower teeth. The angular data can further be used to determine a side of the teeth that is imaged, e.g. a facial side, occlusal side, or lingual side of the teeth.

[0040] In some cases, the motion sensor comprises an inertial measurement unit (IMU). An IMU may comprise an accelerometer and a gyroscope. Accordingly, the IMU can provide both angular data (e.g. as described above) as well data indicative of motion of the oral probe device. This enables accurate determination by the controller, based on outputs from the accelerometer and gyroscope.

[0041] The controller may comprise a machine learning module configured to determine if the oral probe device is in the user’s mouth. The machine learning module may include a first machine learning model configured to use the image data, and / or a second machine learning model configured to use the motion sensor output, for determining if the oral probe device is in the user’ s mouth. This could also include a machine learning model that takes inputs from both the image sensor and the motion sensor. The machine learning module can leverage knowledge gained fromP005258-W00111large volumes of training data to recognise when the oral probe device is inserted into the user’s mouth, thus allowing for an accurate determination across a range of usage scenarios.

[0042] The machine learning module may be adapted to the specific type(s) of sensor(s) used for determining when the oral probe device is in the mouth. As an example, the machine learning module may comprise a classifier model that is configured (e.g. trained) to classify received data (e.g. from the image sensor and / or motion sensor) as corresponding to the oral probe device being inside the mouth or outside the mouth. The machine learning model(s) may be trained, for example, using supervised learning techniques with labelled training data comprising examples of sensor outputs (e.g. image data and / or motion sensor data) corresponding to the oral probe device being inside the mouth or outside the mouth.

[0043] Thus, in some cases, the machine learning module may be configured to determine, based on the image data from the image sensor and the output from the motion sensor, if the oral probe device is in the user’s mouth. Using both the image data and motion sensor data may improve a reliability of a determination of when the oral probe device is in the user’s mouth, in turn providing accurate control of the light source. For example, determinations performed with the image data and the motion sensor data can be used to cross-check one another, to improve reliability and accuracy of the determination.

[0044] The machine learning module could comprise a single model (e.g. neural network) which process both the image data and motion sensor data. Alternatively, separate models may be used for the image data and the motion sensor data, with the outputs of the two models being combined to provide a determination result.

[0045] The controller may be configured to provide, as an input to the machine learning module, input data comprising a combination of features from the image data and the output from the motion sensor. In this manner, a single input may be provided to the machine learning module, which may improve an efficiency of the machine learning module. For example, in line with the above, the machine learning module may comprise a single machine learning model which takes the combined data as an input, and provides a determination result in response.P005258-W00112

[0046] The input data may comprise, for example, a first set of features extracted from the image data, and a second set of features extracted from the output from the motion sensor. The first and second sets of extracted features may be combined in any suitable manner for providing an input to the machine learning module. For example, the first and second sets of extracted features may be combined into a feature vector which is provided as an input to the machine learning module.

[0047] In some cases, the controller may be configured to: generate with the machine learning module a first determination of whether the oral probe device is in the user’ s mouth based on the image data; generate with the machine learning module a second determination of whether the oral probe device is in the user’s mouth based on the output from the motion sensor; and determine, as a function of the first determination and the second determination, if the oral probe device is in the user’s mouth. In line with the above, the first and second determinations could be generated with separate machine learning models, or with a single model that takes both image data and motion sensor data as inputs. The final determination could be performed in any suitable manner, e.g. via a weighting or voting system. This enables, for example, cross-checking between the first determination and the second determination to provide a reliable determination result.

[0048] The oral probe device may comprise a toothbrush, the image sensor and the light source being disposed in or next to a head of the toothbrush. In this manner, the image sensor is arranged to image an area of the user’s mouth that is next to the toothbrush head. For example, a user could image an area they are about brush, or have just brushed, e.g. to check if the area needs cleaning.

[0049] The head of the toothbrush may correspond to a portion of the toothbrush comprising bristles for brushing teeth. The image sensor and the light source may be arranged on a same side of the head as the bristles, e.g. so that the image sensor images an area in front of the bristles.

[0050] The toothbrush may comprise an electric toothbrush, e.g. comprising a motor configured to move the bristles.

[0051] The controller may be configured to: when the oral probe device is determined to be in the user’s mouth, set a brightness of the light source to a first brightness level; and when the oral probe devices is not determined to be in the user’sP005258-W00113mouth, set a brightness of the light source to a second brightness level, the first brightness level being greater than the second brightness level. In this manner, the brightness of the light source is automatically increased when the oral probe device is in the user’s mouth, thus facilitating capturing images inside the mouth. Moreover, the light source is automatically dimmed (or switched off) outside the mouth, to avoid accidentally shining a bright light in the user’s eyes.

[0052] In some cases, the second brightness level may be zero, i.e. such that the light source is switched off outside the user’s mouth, whilst the first brightness level may be non-zero. Accordingly, the controller may be configured to activate (switch on) the light source when the oral probed device is determined to be in the user’s mouth, and to deactivate (switch off) the light source when the oral probe device is determined to be outside the user’s mouth.

[0053] The light source may have a wavelength in a range of 400 to 450 nm. Such a wavelength may cause features of interest such as plaque and / or dentine in the user’s mouth to fluoresce, thus facilitating detection and / or visualisation of the features of interest.

[0054] Additionally or alternatively, the light source may comprise a white light source. This enables capture of white light images of the inside of the user’s mouth.

[0055] In some cases, the controller may be disposed in the oral probe device. In this manner, the analysis of the sensor data (e.g. image data and / or motion sensor data) as well as the control of the light source is performed locally at the oral probe device. This may provide for a more responsive control of the light source in response to movement of the oral probed device.

[0056] In such a case, the controller may be implemented by a suitable processing device or system located in the oral probe device. The controller may comprise a processing unit and a memory storing computer instructions for performing the analysis and control steps described herein.

[0057] Alternatively, the controller may comprise a host device that is separate from the oral probe device. In such a case, the oral probe device may be communicatively coupled (e.g. via a wired and / or wireless connection) to the host device, so that data can be transmitted (e.g. streamed) from the oral probe device to the host device. For example, the oral probe device may be configured to transmit sensor data (e.g. imageP005258-W00114data and / or motion sensor data) to the host device, where the sensor data is processed to determine if the oral probe device is in the mouth. The host device may be configured to transmit, to the oral probe device, control instructions for controlling the brightness of the light source, e.g. based on whether the oral probe device was determined to be in the mouth or not.BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Fig. 1 shows a schematic diagram of an oral probe device;

[0059] Fig. 2 shows a schematic diagram of an oral inspection system;

[0060] Fig. 3 shows a flow diagram of a process performed by an oral inspection system;

[0061] Fig. 4a shows a first example of a machine learning module in an oral inspection system;

[0062] Fig. 4b shows a second example of a machine learning module in an oral inspection system;

[0063] Fig. 4c shows a third example of a machine learning module in an oral inspection system;

[0064] Fig. 5 shows example image data captured with an image sensor in an oral probe device;

[0065] Fig. 6 shows an example output signal from an accelerometer in an oral probe device;

[0066] Fig. 7 shows example output signals from a gyroscope in an oral probe device;

[0067] Fig. 8 shows example output signals from a gyroscope in an oral probe device, corresponding to the oral probe device being at different regions in a user’s mouth; and

[0068] Fig. 9 shows a schematic diagram of different regions in a user’s mouth.DETAILED DESCRIPTION

[0069] Fig. 1 shows a schematic diagram of an oral inspection system comprising an oral probe device 100. The oral probe device 10 is an electric toothbrush for cleaning a user’s mouth. The toothbrush includes a brush head 14 having bristles 12, and aP005258-W00115motor (not shown) for moving the bristles 12 in a brushing motion. The brush head 14 is located at an end of a neck 13 (e.g. elongate shaft) that extends from a handle (or main body) 15 of the toothbrush. The oral probe device 10 further comprises an imaging module 16 including an image sensor 18 and a light source 20. In the example shown, the imaging module 16 is located on the neck 13, next to the head 14. In other examples, the imaging module 16 may be provided in the head 14. The oral probe device 10 further includes a power source (not shown), such as an internal battery, for powering components of the oral probe device 10.

[0070] The image sensor 18 is arranged to image an area of a user’s mouth that is in front of the toothbrush head 14, and the light source 20 is arranged to illuminate the area imaged by the image sensor 18. The image sensor 18 may, for example comprise camera such as a CCD or CMOS camera. The light source 20 may, for example, comprise an LED. The light source 20 may be configured to emit light at a wavelength in a range of 400 to 450 nm, e.g. at 405 nm, to stimulate fluorescence in features of interest in the user’s mouth such as plaque and / or dentine. In some cases, the imaging module 16 may comprise multiple light sources, such as a first light source that emits light at a wavelength in the range of 400 to 450 nm, and a second broadband light source which can be used for white light imaging.

[0071] The oral probe device 10 may further include a motion sensor 22 for detecting a motion of the oral probe device 10. The motion sensor 22 may include an accelerometer and / or a gyroscope. In some cases, the motion sensor 22 may include an inertial measurement unit (IMU), which includes both an accelerometer and a gyroscope. In the example shown, the motion sensor 22 is located in the handle 15, however it may be arranged at other locations in the oral probe device 10.

[0072] The oral probe device 10 further includes an onboard controller 24, which is configured to control operation of the light source 18 as described in more detail below. In particular, the controller 24 can switch the light source 18 on or off, and / or adjust a brightness (intensity) of the light source 18. The onboard controller 24 is further arranged to receive image data (e.g. captured images) from the image sensor 20, and to receive an output signal from the motion sensor 22. The onboard controller 24 may be configured to control operation of the image sensor 20 to capture (acquire) images with the image sensor 20. The onboard controller 24 may be implementedP005258-W00116using any suitable processing unit or device, having software installed thereon for performing the processing steps described herein.

[0073] In some implementations, as shown in Fig. 2, the oral probe device 10 is in communication with a separate host device 26. The host device 26 may, for example, be a smartphone, tablet or laptop computer, or other suitable computing device. The onboard controller 24 may therefore comprise a communication module for wireless communication with the host device 26. For example, the oral probe device 10 may be configured to communicate with the host device 26 via a Bluetooth® connection, via a Wi-Fi network, or other suitable wireless communication method. Additionally or alternatively, the oral probe device 10 may comprise a connector for providing a wired connection between the oral probe device 10 and the host device 26, to enable wired communication.

[0074] Fig. 3 shows a flow diagram of a process 30 performed by an oral inspection system. In some cases, the oral inspection system may be provided by the oral probe device 10 on its own, i.e. all of the steps of process 30 may be performed locally at the oral probe device 10 with the onboard controller 24. Alternatively, as described further below, some of the steps of process 30 may be performed by a software module (e.g. an application) running on the host device 26, in which case the oral inspection system may be considered to include the oral probe device and the host device 26.

[0075] In a first step 32 of the process 30, the onboard controller 24 receives data from one or more sensors in the oral probe device 10. The received data can include image data captured with the image sensor 20, and or data obtained from the motion sensor 22, examples of which are provided below.

[0076] In a second step 34, the oral inspection system determines, as a function of the data received in the first step 32, if the oral probe device is in the user’s mouth. In particular, the oral inspection system determines if imaging module 16 (i.e. the light source 18 and image sensor 20) is located in the user’s mouth. Where the process 30 is performed locally at the oral probe device 10, the onboard controller 24 processes the received sensor data to perform the determination. Where the oral probe device 10 is in communication with the host device 26, the step 34 may be performed by software (e.g. an application) at the host device 26. Accordingly, theP005258-W00117sensor data received in step 32 may be transmitted by the oral probe device 10 to the host device 26, for analysis at the host device 26. Various types of classification algorithm or model may be used for determining if the oral probe device 10 is in the user’s mouth as a function of the received sensor data, examples of which are provided below. The determination in step 34 may be performed using machine learning techniques and / or other data analysis techniques.

[0077] In a third step 36, the oral inspection system controls a brightness of the light source 18 as a function of the determination result in step 34. In general, the light source 18 may be controlled such that it has a greater brightness when the oral probe device 10 (e.g. the imaging module 16) is inside the mouth compared to when the oral probe device is outside the mouth. This may comprise, for example, setting the brightness of the light source 18 to a first, lower brightness level if the oral probe device 10 is determined to be outside the mouth, and setting the brightness of the light source 18 to a second, higher brightness level if the oral probe device 10 is determined to be in the mouth. In some cases, this could include switching off the light source 18 when the oral probe device 10 is outside the mouth, and switching on the light source 18 when the oral probe device 10 is in the mouth.

[0078] Where the process 30 is performed locally at the oral probe device 10, the onboard controller 24 controls the brightness of the light source 18 as a function of the determination result in step 34. For example, the onboard controller 24 may be configured to control an amount of power supplied to the light source 18 (e.g. via suitable power control circuitry), to control the brightness of the light source 18. Where the second step 34 is performed at the host device 26, step 36 may further comprise transmitting control instructions from the host device 26 to the oral probe device 10, where the control instructions are determined based on the determination result in step 34. In response to receipt of the control instructions at the oral probe device 10, the onboard controller 24 controls the brightness of the light source in accordance with the control instructions. For example, the control instructions may comprise an indication of a brightness or power level for the light source 18, or whether the light source 18 should be on or off.

[0079] Figs. 4a-c show schematic diagrams of example machine learning modules 40a-c that can be used for performing the step 34, i.e. for determining if the oralP005258-W00118probe device 10 is in the user’s mouth. Where the process 30 is performed locally at the oral probe device 10, one or more of the machine learning modules 40 may be implemented by the onboard controller 24, e.g. as software executed by the onboard controller 24. Alternatively, where the oral probe device 10 is in communication with the host device 26, the machine learning module(s) 40 may be implemented as software running on the host device 26.

[0080] Starting with Fig. 4a, the machine learning module 40a is configured to receive as an input image data 42 from the image sensor 20. The image data 42 may comprise one or more images captured with the image sensor 20. The image data 42 may correspond to raw data received from the image sensor 20. Alternatively, one or more pre-processing steps may be applied to the data received from the image sensor 20, e.g. to put the data in a format that is adapted to the machine learning module 40a. The machine learning module 40a provides the image data 42 to a machine learning model 46a which is configured to determine, using the image data, if the oral probe device 10 is in the user’s mouth. In particular, the machine learning model 46a is a classifier model that is configured to classify the image data 42 as corresponding to the oral probe device 10 being in the user’s mouth or not. The machine learning model 46a outputs a classification result 48a in response to the input data, which provides the determination for step 34 discussed above.

[0081] The machine learning model 46a comprises an artificial neural network such as a convolutional neural network (CNN) that is trained to classify images as corresponding to the oral probe device 10 being inside the mouth or outside the mouth. A specific example of a CNN that can be used is MobileNetV2, which is a lightweight, efficient architecture ideal for image classification tasks. It can be trained using supervised learning with labelled data (images of the oral probe inside and outside the mouth). The machine learning model 46a can be trained using a supervised learning process, with a set of labelled training data. The training data may comprise images captured inside and outside the mouth, with each image being labelled to indicate whether it corresponds to inside or outside the mouth. The images in the training dataset may, for example, be captured using one or more oral probe devices 10 with a range of different users. In the machine learning model may be adapted using transfer learning, where a pre-trained model (e.g. on ImageNet) can beP005258-W00119fine-tuned to the specific dataset to improve performance, which may be useful if the dataset is limited.

[0082] Fig. 5 shows examples of image data that could be used for training the machine learning model. For instance, panels (a) and (b) of Fig. 5 show images captured inside a user’s mouth with the oral probe device 10, whilst panels (c) and (d) show images captured outside the user’s mouth with the oral probe device 10. Thus, the images in panels (a) and (b) may be labelled as corresponding to inside the mouth, whilst the images in panels (c) and (d) may be labelled as corresponding to outside the mouth. The images in panels (a) and (b) show features such as teeth and gums, which the machine learning model 46a may learn to recognise, in order to determine if an image was captured in the user’s mouth or not. In this manner, when the machine learning model 46a receives new (previously unseen) image data from the image sensor 20, the machine learning model 46a can classify the image data to predict whether the oral probe device 10 is in the mouth or not.

[0083] In some cases, traditional image processing techniques may be used instead of, or in addition to machine learning techniques for determining if the oral probe device 10 is in the user’s mouth based on the image sensor data 42. Such image processing techniques may comprise performing one or more of a colour analysis, a texture analysis, and a shape detection on the image data. For example, Local Binary Patterns (LBP) or Gabor Filters can be for extracting and / or analysing surface textures in the captured images. For shape detection, techniques such as Canny edge detection, contour detection, and Hough Transform can identify the outlines of teeth, gums, and other oral structures in the captured images. Morphological operations and / or the Watershed algorithm can be used to help refine shape segmentation, especially when structures overlap. Additionally or alternatively, colours of the captured images may be analysed by performing a statistical analysis of RGB values or alternative colour spaces like HSV and Lab in the captured images to differentiate between teeth, gums, and background, e.g. based on mean, variance, and histogram distributions. Spatial relationships between colours, such as pink (gums) next to white (teeth), can also be analysed using colour segmentation or co-occurrence matrices to identify structures or objects in the mouth. Such traditional imageP005258-W00120processing techniques may be combined with machine learning techniques to improve accuracy in detecting images corresponding to inside and outside the mouth.

[0084] Turning to Fig. 4b, the machine learning module 40b is configured to receive as an input motion sensor data 44 from the motion sensor 22. The motion sensor data 44 may comprise one or more output signals from the motion sensor 22. For example, where the motion sensor 22 comprises an accelerometer, the motion sensor data 44 may comprise an output signal from the accelerometer, e.g. indicative of an amplitude of an acceleration detected by the accelerometer. Where the motion sensor 22 comprises a gyroscope, the motion sensor data 44 may comprise one or more output signals indicative of an orientation or change in orientation of the oral probe device 10. The angular data may be indicative of one or more of a pitch angle, yaw angle, and roll angle of the oral probe device 10. The motion sensor data 44 may correspond to raw data received from the motion sensor 22. Alternatively, one or more pre-processing steps may be applied to the data received from the motion sensor 22, e.g. to put the data in a format that is adapted to the machine learning module 40b.

[0085] The machine learning module 40b provides the motion sensor data 44 to a machine learning model 46b which is configured to determine, using the motion sensor data 44, if the oral probe device 10 is in the user’s mouth. In particular, the machine learning model 46b is a classifier model that is configured to classify the motion sensor data 44 as corresponding to the oral probe device 10 being in the user’ s mouth or not. The machine learning model 46b outputs a classification result 48b in response to the input data, which provides the determination for step 34 discussed above.

[0086] The machine learning model 46b comprises an artificial neural network such as a convolutional neural network (CNN) that is trained to classify motion sensor data as corresponding to the oral probe device 10 being inside the mouth or outside the mouth. For example, a model such as MobileNetV2 may be used, as mentioned above. The machine learning model 46b can be trained using a supervised learning process, with a set of labelled training data. For example, the machine learning model 46b may be trained using supervised learning algorithms such as Random Forest or XGBoost. The training data may comprise multiple sets of output signals from theP005258-W00121motion sensor 22 that were obtained when oral probe device 10 was inside and outside the mouth, with respective portions of the output signals being labelled as corresponding to inside or outside the mouth. The sets of output signals in the training data may, for example, be recorded during use of one or more oral probe devices 10 by one or more users.

[0087] As noted above, in some implementations, the motion sensor 22 can include an accelerometer. Fig. 6 shows an example of an output signal from the accelerometer. The accelerometer detects vibrational motion in the oral probe device 10 when the motor of the electric toothbrush is switched on. In the example of Fig.6, the motor is switched on, which results in vibrations at a frequency of about 250 Hz being detected by the accelerometer. As shown, the accelerometer signal includes a first portion 60 and a second portion 62, both of which have a higher amplitude than a third portion 64. The third portion 64 of the signal corresponds to the oral probe device 10 being in the user’s mouth, with the reduction in amplitude in the third portion 64 being caused by the bristles 12 of the toothbrush contacting the user’s teeth (e.g. during brushing), which dampens the detected vibrations. On the other hand, first and second portions 60, 62 of the signal correspond to the brush head 14 being outside the user’s mouth, where there may be no (or less) dampening of the vibrations, thus resulting in the higher amplitude of the signal. Accordingly, the first and second portions 60, 62 of the signal may be labelled as corresponding to the oral probe device 10 being outside the user’s mouth, and third portion 64 may be labelled as corresponding to the oral probe device 10 being in the user’s mouth. The machine learning model 46b may thus be trained to recognise that the oral probe device 10 is in the user’s mouth when a reduction in amplitude of the accelerometer signal is observed.

[0088] Step 34 may alternatively be performed without machine learning techniques, e.g. by comparing the amplitude of the vibrations detected by the accelerometer to a predetermined threshold. If the amplitude of the vibrations is below the predetermined threshold, then the system may be configured to determine that the oral probe device 10 is in the user’s mouth. For example, the system may be configured to that the oral probe device 10 is in the user’s mouth if a detected amplitude of the vibration is below 70% (or below 60%) of a predeterminedP005258-W00122maximum amplitude of the vibration. The predetermined maximum amplitude may correspond to an amplitude of the vibration when the oral probe device is outside the user’s mouth, and may be determined experimentally.

[0089] As noted above, in some implementations, the motion sensor 22 can include a gyroscope. Fig. 7 shows an example of angular data obtained from the gyroscope. The angular data comprises a first signal 70 indicative of a yaw angle of the oral probe device 10 as a function of time, a second signal 72 indicative of a roll angle of the oral probe device 10 as a function of time, and a third signal 74 indicative of a pitch angle of the oral probe device 10 as a function of time. The roll angle corresponds to an angle of rotation of the oral probe device 10 about a longitudinal axis of the oral probe device 10. For example, the longitudinal axis may extend in a direction from the handle 15 to the brush head 14. The yaw angle corresponds to an angle of rotation of the oral probe device 10 about a first normal axis that intersects and is normal to the longitudinal axis. The pitch angle corresponds to an angle of rotation of the oral probe device 10 about a second normal axis that intersects and is normal to both the longitudinal axis and the first normal axis. It should be noted that a raw output signal from the gyroscope may be indicative of angular velocity (or rotational speed). The angular data including the yaw, roll and pitch can be derived from the angular velocity output by the gyroscope alone or in combination with accelerometer data, using various known mathematical models (which may be referred to as attitude estimators). Examples of models that can be used for obtaining the angular data from a gyroscope output signal are provided at the following webpage : http s : / / ahrs . readthedoc s . i o / en / 1 atest / filters . html .

[0090] The angular data in Fig. 7 comprises a first portion 76 which corresponds to the oral probe device 10 being inside the user’s mouth, and a second portion 78 which corresponds to the oral probe device 10 being outside the user’s mouth. The first and second portions 76, 78 of the angular data may be labelled accordingly, and used as part of a training dataset for training the machine learning model 46b. In general, there may be greater freedom in terms of possible orientations of the oral probe device 10 outside the mouth compared to when it is in the mouth, e.g. as it may generally be in a predetermined orientation when inserted into the mouth. Accordingly, the machine learning model 46b may be trained to recognise, from theP005258-W00123angular data, orientations of the oral probe device 10 that correspond to it being in the mouth.

[0091] Fig. 8 shows an example of multiple portions 80-95 of angular data obtained from the gyroscope, when the oral probe device 10 is in the mouth. As in Fig. 7, each portion 80-95 of angular data in Fig. 8 includes first, second, and third signals 70, 72, 74 indicative of yaw, roll, and pitch of the oral probe device 10. Each portion SO- 95 of angular data was obtained with the oral probe device 10 arranged to brush a respective region in the mouth, which is indicated in Fig. 9 with the corresponding reference numeral. The regions in the mouth corresponding to the signal portions SO- 95 are as follow:

[0092] 80: Lower Front Facial,

[0093] 81 : Lower Front Lingual,

[0094] 82: Lower Left Facial,

[0095] 83 : Lower Left Lingual,

[0096] 84: Lower Left Occlusal,

[0097] 85: Lower Right Facial,

[0098] 86: Lower Right Lingual,

[0099] 87: Lower Right Occlusal,

[0100] 88: Upper Front Facial,

[0101] 89: Upper Front Lingual,

[0102] 90: Upper Left Facial,

[0103] 91 : Upper Left Lingual,

[0104] 92: Upper Left Occlusal,

[0105] 93 : Upper Right Facial,

[0106] 94: Upper Right Lingual,

[0107] 95: Upper Right Occlusal.

[0108] In order to brush each respective region in the mouth, the oral probe device will typically be held in a respective orientation, which will be reflected in the yaw, roll, and pitch angles of the oral probe device 10. Accordingly, different combinations of yaw, roll, and pitch angles can be associated with the oral probe device 10 being arranged to brush different regions in the mouth. Thus, in some implementations, the portions 80-95 of angular data can be labelled to indicate theP005258-W00124corresponding region in the mouth. In this manner, the machine learning model 46b can be trained to determine if the oral probe device 10 is in the mouth, and which region of the mouth is in front of the brush head 14.

[0109] It should be noted that machine learning techniques need not necessarily be used to determine if the oral probe device 10 is in the mouth based on the angular data. For example, the system may be configured to determine that the oral probe device 10 is in the mouth if the oral probe device is in a predetermined orientation. The predetermined orientation may, for example, be defined as a respective range of angles for each of the yaw, roll, and pitch of the oral probe device 10.

[0110] Returning to Fig. 4b, where the motion sensor 22 comprises both the accelerometer and the gyroscope, the machine learning model 46b may be configured to receive the signals from both the accelerometer and the gyroscope as inputs. The machine learning model 46b may then be configured to determine whether the oral probe device 10 is in the mouth based on the combination of data from the accelerometer and the gyroscope. The machine learning model 46b may thus be trained using a combination of labelled accelerometer and gyroscope data, in a manner analogous to that described above. Alternatively, the machine learning model 46b may be implemented as two sperate machine learning models, each of which is configured to deal with the accelerometer data and gyroscope data, respectively. Respective determination results may then be produced based on each of the accelerometer data and the gyroscope data. The respective determination results may then be combined, e.g. using a voting system or weighting system to produce a combined determination for the motion sensor data.

[0111] Fig. 4c shows an example machine learning module 40c which is configured to receive both the motion sensor data 44 and the image data 42 as inputs. The motion sensor data 44 and the image data 42 may be as described above. The machine learning module 40c provides the motion sensor data 44 and the image data 42 to a machine learning model 46c which is configured to determine, using the received data, if the oral probe device 10 is in the user’s mouth. In some cases, as shown, the machine learning model 46c may be composed of the machine learning model 46a and the machine learning model 46b, described above. The image data may then be provided as an input to the machine learning model 46a which provides a firstP005258-W00125classification result 48a based on the image data, and the motion sensor data 44 may be provided as an input to the machine learning model 46b which provides a second classification result 48b. The first classification result 48a and the second classification result 48b may then be combined to provide an output classification result 48c for the machine learning module 40c. As an example, predetermined weightings may be applied to the first classification result 48a and the second classification result 48b, to determine the classification result 48c. For instance, if the image classification model 46a is deemed more reliable due to higher accuracy or better data quality, a higher weight (e.g., 0.7) can be applied to the image classification result, and a lower weight (e.g., 0.3) to the motion sensor classification result 48b). The final classification result 48c may then be computed as a weighted sum or average of the two results. For example: Classification Result 48c = wl x Result 48a + w2 * Result 48b, where wl and w2 are predetermined weightings.

[0112] As another example, the machine learning module 40c may implement a voting system for deciding the output classification result 48c based on the first and second classification results 48a, 48b. For instance, if the first machine learning model 46a and the second machine learning model 46b both provide binary classifications (e.g., "inside" or "outside"), the classification with the majority vote becomes the final decision. In cases of a tie, additional logic (e.g., selecting based on which classifier has higher confidence) can be used to break the tie. This method allows for a more flexible and adaptive combination of the two data sources, ensuring the final classification benefits from both inputs.

[0113] The examples described above are illustrative of the present disclosure, and further examples are envisaged. It is to be understood that any feature described in relation to any one example may be used alone or in combination with other features of the example, and may also be used in combination with one or more features of any other of the examples, or any combination of any other of the examples. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the disclosure, which is defined in the accompanying claims.

Claims

P005258-W00126CLAIMS1. An oral inspection system comprising:an oral probe device for insertion into a user’s mouth, the oral probe device comprising an image sensor, and a light source for illuminating an area imaged by the image sensor; anda controller configured to:determine if the oral probe device is in the user’s mouth; and control a brightness of the light source as a function of the determination.

2. An oral inspection system according to claim 1, wherein the controller is configured to receive image data from the image sensor, and to determine if the oral probe device is in the user’ s mouth using the image data.

3. An oral inspection system according to claim 2, wherein the controller is configured to perform a colour analysis and / or a texture analysis of the image data to determine if the oral probe device is in the user’s mouth.

4. An oral inspection system according to claim 2 or 3, wherein the controller is configured to perform shape detection on the image data to determine if the oral probe device is in the user’s mouth.

5. An oral inspection system according to any preceding claim, wherein the oral probe device further comprises a motion sensor for detecting a motion of the oral probe device, and wherein the controller is configured to receive an output from the motion sensor and determine if the oral probe device is in the user’s mouth using the output from the motion sensor.

6. An oral inspection system according to claim 5, wherein the motion sensor is configured to detect a vibrational motion of the oral probe device, and wherein the controllerP005258-W00127is configured to determine if the oral probe device is in the user’s mouth based on an amplitude of the vibrational motion.

7. An oral inspection system according to claim 5 or 6, wherein the motion sensor is configured to detect an orientation of the oral probe device, and wherein the controller is configured to determine if the oral probe device is in the user’s mouth based on the detected orientation.

8. An oral inspection system according to claim 7, wherein the motion sensor is configured to output angular data indicative of at least two of a pitch, roll, and yaw of the oral probe device, and wherein the controller is configured to determine if the oral probe device is in the user’s mouth as a function of the angular data for each of the at least two of the pitch, roll, and yaw.

9. An oral inspection system according to any of claims 5 to 8, wherein the motion sensor comprises an inertial measurement unit.

10. An oral inspection system according to any preceding claim, wherein the controller comprises a machine learning module configured to determine if the oral probe device is in the user’s mouth.

11. An oral inspection system according to claim 10 as dependent on claims 2 and 5, wherein the machine learning module is configured to determine, based on the image data from the image sensor and the output from the motion sensor, if the oral probe device is in the user’s mouth.

12. An oral inspection system according to claim 11, wherein the controller is configured to provide, as an input to the machine learning module, input data comprising a combination of features from the image data and the output from the motion sensor.

13. An oral inspection system according to claim 11 or 12, wherein the controller is configured to:P005258-W00128generate with the machine learning module a first determination of whether the oral probe device is in the user’s mouth based on the image data;generate with the machine learning module a second determination of whether the oral probe device is in the user’s mouth based on the output from the motion sensor; anddetermine, as a function of the first determination and the second determination, if the oral probe device is in the user’s mouth.

14. An oral inspection system according to any preceding claim, wherein the oral probe device comprises a toothbrush, the image sensor and the light source being disposed in or next to a head of the toothbrush.

15. An oral inspection system according to any preceding claim, wherein the controller is configured to:when the oral probe device is determined to be in the user’s mouth, set a brightness of the light source to a first brightness level; andwhen the oral probe device is not determined to be in the user’s mouth, set a brightness of the light source to a second brightness level, the first brightness level being greater than the second brightness level.

16. An oral inspection system according to any preceding claim, wherein the light source has a wavelength in a range of 400 to 450 nm.

17. An oral inspection system according to any preceding claim, wherein the controller is disposed in the oral probe device.

Images