Toothbrush system, control method, and program

Abstract

This toothbrush system comprises: a toothbrush that cares for the inside of the oral cavity; a light source that emits, to a care site including a tooth area, light having a prescribed wavelength with which plaque reacts; an imaging unit that acquires, during brushing by using the toothbrush, a plurality of first RGB images including the care site, and fluorescence and reflected light from the care site; a first image generation unit (72) that generates a plurality of second RGB images by performing, on the plurality of first RGB images, image processing that achromatizes a region of a tooth; a second image generation unit (75) that detects the presence or absence of plaque in the tooth area on the basis of an evaluation image included in the plurality of second RGB images at a timing corresponding to a user operation during brushing, and generates a plaque-highlighted image indicating the presence or absence of plaque in the tooth area; and an output unit (for example, a display unit (76)) that outputs the plaque-highlighted image.

Description

Toothbrush system, control method, and program

[0001] The present disclosure relates to a toothbrush system, a control method, and a program.

[0002] Conventionally, an electric toothbrush that operates with vibrations suitable for removing dental plaque has been proposed. There is a demand from users of electric toothbrushes to present information such as where dental plaque is attached and whether the dental plaque has been removed by brushing. In order to meet this demand, in Patent Document 1, the posture of a care member such as a toothbrush for oral care is detected to estimate the oral care site, and light of a predetermined wavelength to which dental plaque reacts is irradiated onto the teeth, and the amount of dental plaque is detected from the image data of the reflected light from the care site. An oral care device has been proposed. According to the oral care device, the care site and the amount of dental plaque can be associated and stored in a memory and presented to the user.

[0003] Japanese Patent Application Laid-Open No. 2013-042906

[0004] However, in the device of Patent Document 1, the user can only confirm the oral care site and the amount of dental plaque detected corresponding to the care site after brushing. That is, the device of Patent Document 1 cannot assist the user during brushing.

[0005] Therefore, the present disclosure provides a toothbrush system, a control method, and a program that can assist the user during brushing.

[0006] A toothbrush system according to an aspect of the present disclosure includes a toothbrush for oral care, a light source that emits light of a predetermined wavelength to which dental plaque reacts to a care site including a tooth region, and during brushing with the toothbrush, a photographing unit that acquires a plurality of first RGB images including the care site, reflected light, and fluorescence from the care site, a first image generation unit that generates a plurality of second RGB images by performing image processing to desaturate the teeth in the plurality of first RGB images, a second image generation unit that detects the presence or absence of dental plaque in the tooth region based on an evaluation image included in the plurality of second RGB images at a timing corresponding to the operation of the user during brushing and generates a dental plaque emphasized image indicating the presence or absence of dental plaque in the tooth region, and an output unit that outputs the dental plaque emphasized image.

[0007] A control method according to one aspect of the present disclosure is a control method for a toothbrush system, the toothbrush system comprising: a toothbrush for caring for the oral cavity; a light source that emits light of a predetermined wavelength to which plaque reacts toward a care area including a tooth region; and an imaging unit that acquires a plurality of first RGB images including the care area and reflected light and fluorescence from the care area while brushing with the toothbrush, the control method generates a plurality of second RGB images by performing image processing on the plurality of first RGB images to make the teeth grayscale; detects the presence or absence of plaque in the tooth region based on evaluation images included in the plurality of second RGB images at a timing corresponding to the user's operation during brushing; generates a plaque-enhanced image indicating the presence or absence of plaque in the tooth region; and outputs the plaque-enhanced image.

[0008] A program relating to one aspect of this disclosure is a program that causes a computer to execute the control method described above.

[0009] According to one aspect of this disclosure, a toothbrush system or the like that can be realized to assist the user while brushing their teeth.

[0010] Figure 1 is an external view of the main body of the toothbrush system according to the embodiment. Figure 2 is an enlarged view of the head according to the embodiment. Figure 3 is an external view when the toothbrush part according to the embodiment is attached to the main body. Figure 4 is an external view when the toothbrush part according to the embodiment is attached to the main body. Figure 5 is an external view when the light-shielding member according to the embodiment is attached to the main body. Figure 6 is a schematic diagram showing an example of use of the toothbrush system according to the embodiment. Figure 7 is a schematic configuration diagram of the toothbrush system according to the embodiment. Figure 8 is a functional block diagram of the mobile terminal according to the embodiment. Figure 9 is a flowchart showing the operation flow of the mobile terminal according to the embodiment. Figure 10 is a diagram showing an example of a live image according to the embodiment. Figure 11 is a diagram showing an example of a notification image according to the embodiment. Figure 12 is a diagram showing an example of a plaque-enhancing image according to the embodiment.

[0011] A toothbrush system according to a first aspect of the present disclosure comprises a toothbrush for oral care, a light source that emits light of a predetermined wavelength to which plaque reacts onto a care area including a tooth region, an imaging unit that acquires a plurality of first RGB images including the care area and reflected and fluorescent light from the care area while brushing with the toothbrush, a first image generation unit that generates a plurality of second RGB images by performing image processing on the plurality of first RGB images to make the teeth grayscale, a second image generation unit that detects the presence or absence of plaque in the tooth region based on evaluation images included in the plurality of second RGB images at a timing corresponding to the user's operation during brushing, and generates a plaque-enhanced image showing the presence or absence of plaque in the tooth region, and an output unit that outputs the plaque-enhanced image.

[0012] This allows the toothbrush system to output plaque-enhanced images at a timing corresponding to the user's actions during brushing. In other words, the user can check the plaque-enhanced image output at an effective timing and then brush their teeth using the image as a reference. Thus, it is possible to realize a toothbrush system that can assist the user during brushing. Such a toothbrush system can prevent teeth from being brushed excessively. Furthermore, since the toothbrush system detects plaque using images captured by the imaging unit, it can detect plaque in the tooth area while the user is brushing their teeth.

[0013] Furthermore, for example, the toothbrush system according to the second embodiment is the toothbrush system according to the first embodiment, further comprising a determination unit for determining the stationary state of the toothbrush, the user's operation including the user stopping the toothbrush during brushing, and the timing may include the timing at which the determination unit determines that the toothbrush has stopped during brushing.

[0014] This makes it possible to generate plaque-enhanced images from images taken while the device is stationary, improving the accuracy of plaque detection. Such plaque-enhanced images can effectively assist users in brushing their teeth.

[0015] Furthermore, for example, the toothbrush system according to the third embodiment is a toothbrush system according to the first or second embodiment, and the output unit may output a notification image to notify that the plaque-enhanced image has been generated when the determination unit determines that the toothbrush has come to a stop during brushing and the generation of the plaque-enhanced image has been completed.

[0016] This allows the user to be notified when the generation of the plaque-enhanced image is complete.

[0017] Furthermore, for example, the toothbrush system according to the fourth embodiment is a toothbrush system according to any of the first to third embodiments, wherein the output unit outputs the plaque-enhancing image during the period that the determination unit determines to be the stationary state, and outputs video including the plurality of second RGB images during other periods during brushing.

[0018] This allows for the generation of plaque-enhanced images from still images, improving the accuracy of plaque detection. Furthermore, users can review both the plaque-enhanced image and video while brushing their teeth, enabling more effective brushing. Therefore, this provides better support to users during brushing.

[0019] Furthermore, for example, the toothbrush system according to the fifth embodiment is a toothbrush system according to any of the second to fourth embodiments, and the determination unit may determine that the toothbrush is in a stationary state if, in two or more consecutive second RGB images among the plurality of second RGB images, the movement of the toothbrush region within the image is less than a predetermined first threshold.

[0020] This allows the system to determine whether the toothbrush is stationary or not using an image. In other words, it does not use any information other than the image to determine whether the toothbrush is stationary or not. Therefore, since no additional sensors are needed to determine whether the toothbrush is stationary or not, the complexity of the toothbrush system can be suppressed.

[0021] Furthermore, for example, the toothbrush system according to the sixth embodiment is a toothbrush system according to any of the first to fifth embodiments, wherein the user operation includes an operation to display the plaque-enhancing image generated by the second image generation unit, and the output unit, when the operation is detected, outputs the plaque-enhancing image for a predetermined period of time, and after the predetermined period has elapsed, switches to a video including the plurality of second RGB images and outputs the video.

[0022] This allows users to view plaque-enhanced images and videos. Since users can review these images and videos while brushing their teeth, they can brush more effectively. Therefore, this further assists users during the brushing process.

[0023] Furthermore, for example, the toothbrush system according to the seventh embodiment is a toothbrush system according to any of the first to sixth embodiments, and the second image generation unit may acquire as the evaluation image a second RGB image from among the plurality of second RGB images in which the proportion of pixels constituting natural teeth within the field of view is equal to or greater than the second threshold.

[0024] This improves the accuracy of plaque detection.

[0025] A control method according to an eighth aspect of the present disclosure is a control method for a toothbrush system, the toothbrush system comprising: a toothbrush for oral care; a light source that emits light of a predetermined wavelength to which plaque reacts toward a care area including a tooth region; and an imaging unit that acquires a plurality of first RGB images including the care area and reflected light and fluorescence from the care area while brushing with the toothbrush, the control method generates a plurality of second RGB images by performing image processing on the plurality of first RGB images to make the teeth grayscale; detects the presence or absence of plaque in the tooth region based on evaluation images included in the plurality of second RGB images at a timing corresponding to the user's operation during brushing; generates a plaque-enhanced image showing the presence or absence of plaque in the tooth region; and outputs the plaque-enhanced image.

[0026] This produces the same effect as the toothbrush system described above.

[0027] The program relating to the ninth aspect of this disclosure is a program that causes a computer to execute the control method relating to the eighth aspect.

[0028] This produces the same effect as the toothbrush system described above.

[0029] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.

[0030] The embodiments will be described in detail below with reference to the drawings.

[0031] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, any components in the following embodiments that are not described in an independent claim will be described as optional components.

[0032] Furthermore, each figure is a schematic diagram and not necessarily a strictly accurate representation. Therefore, for example, the scale may not necessarily match in each figure. Also, in each figure, substantially identical components are given the same reference numerals, and redundant explanations may be omitted or simplified.

[0033] Furthermore, in this specification, numerical values ​​and numerical ranges do not represent only strict meanings, but also include substantially equivalent ranges, such as differences of a few percent (or about 10%).

[0034] Furthermore, in this specification, ordinal numbers such as "first," "second," etc., do not mean the number or order of components unless otherwise specified, but are used to avoid confusion and to distinguish similar components.

[0035] Furthermore, in this specification, when we describe, for example, a value greater than or equal to a threshold and a value less than a threshold in comparison, it means that the value is distinguished by that threshold, and may mean that the value is greater than the threshold and less than or equal to the threshold, respectively.

[0036] (Embodiment) [1. Configuration] [1-1. Configuration of the main body] Figure 1 is an external view of the main body 10 in the toothbrush system according to this embodiment. As shown in Figure 1, the main body 10 has a toothbrush-shaped housing that can be handled with one hand, and the housing includes a head portion 10a that is placed in the user's oral cavity when taking dental arch images, a handle portion 10b that the user grasps, and a neck portion 10c that connects the head portion 10a and the handle portion 10b. An operation button 10b1 is provided on the handle portion 10b. The operation button 10b1 may be, for example, a button for performing operations on the imaging unit (for example, an imaging instruction).

[0037] Figure 2 is an enlarged view of the head unit 10a shown in Figure 1. The head unit 10a comprises an imaging optical system 12 (also called the imaging unit) and a plurality of LEDs 26A to 26D.

[0038] For example, the imaging optical system 12 is incorporated into the head portion 10a and the neck portion 10c. For example, the imaging optical system 12 includes an image sensor and a lens arranged on its optical axis.

[0039] The image sensor is an imaging device such as a CMOS (Complementary Metal-Oxide-Semiconductor) sensor or a CCD (Charge Coupled Device) element.

[0040] The imaging optical system 12 may also include a blue light cut filter that cuts out the blue wavelength light component contained in the light incident on the image sensor.

[0041] The plurality of LEDs 26A to 26D are an illumination device (light source) that irradiates light onto the teeth of the subject during imaging. The plurality of LEDs 26A to 26D are, for example, blue LEDs (Light Emitting Diodes) that irradiate blue light having a wavelength with a peak at 405 nm. Also, as shown in FIG. 2, in the case of the present embodiment, the plurality of LEDs 26A to 26D are arranged so as to surround the entrance of the imaging optical system 12. Note that at least a part of the plurality of LEDs 26A to 26D may be white LEDs. Also, an arbitrary light source may be used instead of the LEDs.

[0042] [1-2. Attachment] The toothbrush system according to the present embodiment has one or more attachments that are detachable from the main body 10. Also, the toothbrush system has an oral camera mode for imaging the state inside the oral cavity and a toothbrushing mode for displaying the state inside the oral cavity while brushing teeth.

[0043] In the toothbrushing mode, a toothbrushing part 11a, which is one of the attachments, is attached to the main body 10. FIG. 3 is an external view when the toothbrushing part 11a according to the present embodiment is attached to the main body 10. As shown in FIG. 3, the toothbrushing part 11a is attached to the head part 10a of the main body 10. Note that in FIG. 3, an example of the toothbrushing part 11a having a circular brush is shown, but a toothbrushing part having another shape may be used.

[0044] FIG. 4 is an external view when the toothbrushing part 11b according to the present embodiment is attached to the main body 10. As shown in FIG. 4, a toothbrushing part 11b having a deck brush-shaped brush may be used.

[0045] Note that either one of the toothbrushing part 11a and the toothbrushing part 11b may be used, or both the toothbrushing part 11a and the toothbrushing part 11b may be included in a plurality of attachments of the toothbrush system.

[0046] Also, in the oral camera mode, a light-shielding member may be attached to the head portion 10a in order to reduce the influence of external light. FIG. 5 is an external view when the light-shielding member 11c according to the present embodiment is attached to the main body portion 10. As shown in FIG. 5, by attaching the light-shielding member 11c, which is one of the attachments, to the head portion 10a, the influence of external light can be reduced. As a result, the quality of the captured image is improved, and the accuracy of the processing using the image can be improved. In the oral camera mode, any one of the attachments may be attached to the main body portion 10, or none of the attachments may be attached to the main body portion 10.

[0047] Note that, regardless of which attachment shown in FIGS. 3 to 5 is attached, the imaging optical system 12 and the plurality of LEDs 26A to 26D are exposed, and light can be irradiated onto and an image can be captured of a care site (e.g., teeth) in the oral cavity. Instead of the imaging optical system 12 and the plurality of LEDs 26A to 26D being exposed, they may be covered with a translucent member.

[0048] Note that the shapes of the attachments shown in FIGS. 3 to 5 are merely examples, and attachments of any shape may be used.

[0049] [1-3. Overall Functional Configuration] FIG. 6 is a diagram schematically showing an example of use of the toothbrush system according to the present embodiment. The toothbrush system according to the present embodiment is a system that supports the user U in brushing their teeth by displaying an image of the user U's oral cavity during toothbrushing. For example, the toothbrush system according to the present embodiment provides the user with image information on the state of plaque adhesion on the teeth by capturing a care site with the imaging unit during toothbrushing and highlighting the plaque. Thereby, it is possible to provide the user with information that can assist in accurately brushing their teeth.

[0050] As shown in Figure 6, the toothbrush system displays images of the user U's oral cavity, captured by the imaging unit of the main unit 10, on the display unit 76 of the mobile terminal 70 in real time while the user U is brushing their teeth. Real time means displaying the image to the user U while they are brushing their teeth within the time lag required to assist with brushing. A shorter time lag is desirable, but it does not need to be perfectly simultaneous. Real time may include time lags due to communication via the network, time lags due to the processing speed of devices such as the mobile terminal 70, etc.

[0051] User U can brush their teeth while viewing real-time images of their teeth. Furthermore, by displaying images that include, for example, plaque, User U can brush their teeth more effectively. The toothbrush system can further assist User U in brushing effectively by displaying plaque, such as areas that were missed during brushing.

[0052] Furthermore, user U is not limited to holding the mobile device 70 in their hand while brushing their teeth; they may also brush their teeth with the mobile device 70 placed on a table or other surface.

[0053] Figure 7 is a schematic diagram of the toothbrush system according to this embodiment. As shown in Figure 7, the toothbrush system according to this embodiment is generally configured to use the main body 10 to photograph the teeth and to perform image processing on the captured image.

[0054] As shown in Figure 7, the toothbrush system includes a main unit 10, a mobile terminal 70, and a cloud server 80. The mobile terminal 70 is, for example, a wirelessly wireless smartphone or tablet. The mobile terminal 70 includes a display unit 76, such as a touchscreen capable of displaying dental images, as an input and output device. The mobile terminal 70 functions as the user interface for the toothbrush system.

[0055] The cloud server 80 is a server that can communicate with the mobile terminal 70 via the internet or the like, and provides the mobile terminal 70 with an application for using the main unit 10. For example, the user downloads the application from the cloud server 80 and installs it on the mobile terminal 70. The cloud server 80 also acquires dental images captured by the main unit 10 via the mobile terminal 70.

[0056] The main unit 10 includes a central control unit 50, which is the main part that controls the system, a plurality of LEDs 26A to 26D, an LED control unit 54, an actuator 36, an actuator 40, a lens driver 56, and a position sensor 90.

[0057] Furthermore, the main unit 10 includes a wireless communication module 58 that communicates wirelessly with the mobile terminal 70, and a power supply control unit 60 that supplies power to the central control unit 50 and the like.

[0058] The central control unit 50 of the main unit 10 is mounted, for example, on the handle portion 10b of the main unit 10. For example, the central control unit 50 also includes a controller 62 such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) that executes various processes described later, and a memory 64 such as a RAM (Random Access Memory) or ROM (Read Only Memory) that stores programs for causing the controller 62 to execute various processes. In addition to programs, the memory 64 also stores dental images (image data) captured by the image sensor 14 and various setting data.

[0059] The controller 62 transmits the dental image output from the image sensor 14 to the mobile terminal 70 via the wireless communication module 58. The mobile terminal 70 displays the transmitted dental image on its touchscreen, thereby presenting the dental image to the user.

[0060] The LED control unit 54 is mounted, for example, on the handle portion 10b of the main body 10, and controls the lighting and extinguishing of the multiple LEDs 26A to 26D based on control signals from the controller 62. The LED control unit 54 is composed of, for example, a circuit.

[0061] The actuator 36 is mounted, for example, on the head portion 10a or the neck portion 10c of the main body 10, and adjusts the composition of the imaging optical system 12. Specifically, the actuator 36 moves the housing that holds the image sensor 14, etc., in the direction of extension of the optical axis in order to adjust the field of view.

[0062] The actuator 40 is mounted, for example, on the head portion 10a or the neck portion 10c of the main body 10, and adjusts the focus of the imaging optical system 12. Specifically, the actuator 40 moves the lens in the direction of extension of the optical axis to adjust the focus.

[0063] The lens driver 56 is mounted, for example, on the handle portion 10b of the main body 10, and controls the actuators 36 and 40 based on control signals from the controller 62 of the central control unit 50. The lens driver 56 is composed of, for example, a circuit.

[0064] The wireless communication module 58 is mounted, for example, on the handle portion 10b of the main unit 10 and communicates wirelessly with the mobile terminal 70 based on control signals from the controller 62. The wireless communication module 58 performs wireless communication with the mobile terminal 70 in accordance with existing communication standards such as Wi-Fi® or Bluetooth®. Dental images are transmitted from the main unit 10 to the mobile terminal 70 via the wireless communication module 58. Operation signals are also transmitted from the mobile terminal 70 to the main unit 10 via the wireless communication module 58.

[0065] In this embodiment, the power control unit 60 is mounted on the handle portion 10b of the main body 10 and distributes power from the battery 66 to the central control unit 50, LED control unit 54, lens driver 56, and wireless communication module 58. The power control unit 60 is composed of, for example, a circuit. In this embodiment, the battery 66 is a rechargeable secondary battery and is wirelessly charged by an external charger 69 connected to a commercial power source via a coil 68 mounted on the main body 10.

[0066] The position sensor 90 is a sensor for detecting the attitude and position of the main body 10, and is, for example, a multi-axis (in this case, three axes: x, y, and z) accelerometer. For example, the position sensor 90 may be a six-axis sensor having a three-axis accelerometer and a three-axis gyroscope.

[0067] [1-4. Functional Configuration of the Mobile Terminal] Figure 8 is a functional block diagram of the mobile terminal 70 according to this embodiment. The mobile terminal 70 comprises an acquisition unit 71, a first image generation unit 72, a reception unit 73, a determination unit 74, a second image generation unit 75, and a display unit 76.

[0068] The acquisition unit 71 acquires multiple images transmitted from the main unit 10. The acquired images are, for example, RGB images represented by the RGB color model using the three primary colors R (Red), G (Green), and B (Blue). The first image generation unit 72 generates multiple images that have undergone predetermined image processing by performing predetermined image processing on the multiple images. The predetermined image processing is, for example, white balance processing, but is not limited to this. The reception unit 73 receives operations from the user. The determination unit 74 determines the stationary state of the toothbrush.

[0069] The second image generation unit 75 performs plaque detection processing on at least one image included in a plurality of images that have undergone predetermined image processing, and generates a plaque-enhanced image that includes the plaque detection result. The display unit 76 displays the plurality of images generated by the first image generation unit 72 and the plaque-enhanced image generated by the second image generation unit 75. The display unit 76 is implemented by a display device such as a touchscreen, but is not limited thereto. The display unit 76 is an example of an output unit.

[0070] The functions of the acquisition unit 71, the first image generation unit 72, the reception unit 73, the determination unit 74, the second image generation unit 75, and the display unit 76 are realized, for example, by a processor such as a CPU in the mobile terminal 70 executing a program stored in the memory of the mobile terminal 70. At least a part of the acquisition unit 71, the first image generation unit 72, the reception unit 73, the determination unit 74, the second image generation unit 75, and the display unit 76 may be composed of circuits.

[0071] Furthermore, the mobile terminal 70 may be equipped with notification-capable components such as a speaker, a light source, or a vibrator.

[0072] The mobile terminal 70 may also include a transmitting unit that transmits images to another display device, either together with the display unit 76 or in place of the display unit 76. In other words, the mobile terminal 70 may display plaque-enhanced images or the like on another device. The transmitting unit is an example of an output unit. The transmitting unit may, for example, include a communication circuit.

[0073] [2. Operation] Next, the operation of the toothbrush system configured as described above will be explained with reference to Figures 9 to 12. Figure 9 is a flowchart showing the flow of operation (control method) of the portable terminal 70 according to this embodiment. Note that all of the operations shown in Figure 9 are performed while brushing teeth.

[0074] As shown in Figure 9, first, the acquisition unit 71 acquires a plurality of images transmitted from the main unit 10 (S101). These plurality of images are obtained, for example, by a user taking pictures of their teeth and gums in their oral cavity while performing a toothbrushing action using the main unit 10. These plurality of images are also obtained by the main unit 10 irradiating the teeth with light including the wavelength range of blue light and taking pictures of teeth that are showing a fluorescence reaction. For example, the plurality of images are obtained when light of a predetermined wavelength that plaque reacts to is emitted from the light source to the care area including the tooth region, and the imaging unit takes pictures of the care area being brushed with a toothbrush, as well as multiple images including reflected light and fluorescence from the care area. In addition to teeth, the images may also include gums, gingiva, etc., in the care area. Here, blue light is an example of irradiation light of a predetermined wavelength that excites fluorescent substances contained in plaque. The fluorescent substance is, for example, porphyrin. Porphyrin is a substance produced by bacteria in plaque.

[0075] Here, the multiple images may be a video or multiple still images. Image data may be transmitted for each frame of the video or for each still image, or image data may be transmitted for every multiple frames. For example, the acquisition unit 71 may acquire a video (first video) containing multiple images. The acquisition unit 71 may also acquire sensor data in addition to image data from the main unit 10. The image acquired in step S101 is an example of an RGB image (first RGB image). The first video can also be said to be composed of multiple first RGB images.

[0076] Next, the first image generation unit 72 performs predetermined image processing on multiple images in order to generate a live image to be displayed to the user (S102). The predetermined image processing is a process to dechromatize teeth appearing in a plurality of bluish images, for example, white balance processing. The predetermined image processing may also be a process to adjust the gain of at least two color components among the red, green, and blue components of the RGB image to be processed so that the first average value of the red pixel values ​​of a plurality of first pixels constituting the tooth region of the plurality of images to be processed approaches (for example, becomes equal to) the first average value of the green pixel values ​​of a plurality of first pixels and the first average value of the blue pixel values ​​of a plurality of first pixels. Here, the tooth region may be the region of natural teeth. The image after the predetermined processing is an example of an RGB image (second RGB image).

[0077] Furthermore, the specified image processing may be a process that corrects the color of the natural teeth in the image to match the actual color of the user's natural teeth, based on reference color data of the user's natural teeth acquired in advance.

[0078] Next, the display unit 76 displays multiple images that have undergone predetermined image processing as a live image (S103). It can also be said that the display unit 76 displays the images that have undergone predetermined image processing in real time. This allows the user to brush their teeth while checking the image of the inside of their mouth during brushing. The live image may be a video (second video) containing multiple images that have undergone predetermined image processing.

[0079] Figure 10 shows an example of a live image P1 according to this embodiment. The live image P1 is a display image shown by the display unit 76 while brushing teeth.

[0080] As shown in Figure 10, the live image P1 includes the tooth region 201 (tooth) and the brush portion of the toothbrush 11a. Since the live image P1 is an image that has been dechromatized, the user can see an image that includes teeth in a color close to the actual color of teeth. The live image P1 is an image that has not undergone plaque detection processing, and therefore no plaque is shown. Note that the brush portion does not necessarily have to be visible in the live image P1.

[0081] Referring again to Figure 9, the determination unit 74 determines whether or not it has detected a predetermined operation from the user (S104). The determination unit 74 determines that it has detected a predetermined operation if the receiving unit 73 has received a predetermined operation corresponding to displaying a plaque-enhanced image showing the plaque area.

[0082] The predetermined operation may be stopping the toothbrush while brushing (stopping brushing), pressing the operation button 10b1, touching the mobile terminal 70, or specifying the type of tooth to be checked for plaque. The timing when the toothbrush stops during brushing (for example, the timing when the determination unit 74 determines that the toothbrush has stopped during brushing), the timing when the user presses the operation button 10b1, the timing when the user touches the mobile terminal 70, and the timing when the operation to specify the type of tooth is performed (for example, the timing when the reception unit 73 receives input of the type of tooth) are examples of predetermined timings.

[0083] To stop the toothbrush means to keep the main body 10 still while the head 10a is inside the oral cavity during brushing. Whether or not the toothbrush is stopped is determined by the determination unit 74. For example, the determination unit 74 may determine that the toothbrush is in a stopped state if, in two or more consecutive images from a plurality of images that have undergone predetermined image processing, the movement of the toothbrush region within the images is below a predetermined threshold, or it may determine whether or not the toothbrush is in a stopped state by other means.

[0084] The type of tooth may be identified by any known method. For example, each of the user's teeth may be photographed in advance, stored in association with the type of tooth, and the type of tooth may be identified by comparing the tooth shown in the image from the main unit 10 with the stored image.

[0085] If the second image generation unit 75 determines that a predetermined operation has been detected by the determination unit 74 (Yes in S104), it proceeds to step S105, and the processing from step S105 onwards is performed. In other words, the processing from step S105 onwards is performed at a timing corresponding to the user's operation during brushing. If the determination unit 74 determines that a predetermined operation has not been detected (No in S104), it returns to step S103, and the processing from step S103 onwards continues to be executed.

[0086] Next, if the determination unit 74 determines that a predetermined operation has been detected (Yes in S104), the second image generation unit 75 selects an evaluation image, which is the target image for plaque detection (S105). For example, the second image generation unit 75 detects tooth regions in each of the multiple images and detects the movement of those tooth regions. The second image generation unit 75 may, for example, select an image from among the multiple images in which the movement of the tooth regions is less than a predetermined threshold as the evaluation image. Alternatively, the second image generation unit 75 may, for example, select an image taken when the toothbrush is stationary as the evaluation image. The evaluation image is, for example, a still image.

[0087] This allows for the detection of plaque using images with minimal movement, thereby improving the accuracy of plaque detection. Furthermore, the second image generation unit 75 may select the most recent image at the time of detection of a predetermined operation, or an image acquired immediately after detection of the predetermined operation, as the evaluation image. The second image generation unit 75 functions as a selection unit.

[0088] In step S105, the second image generation unit 75 may, instead of using an image with little motion, select an image as the evaluation image in which the proportion of tooth regions within the image (for example, the proportion of pixels constituting teeth to the image) is equal to or greater than a predetermined threshold. For example, the second image generation unit 75 may detect tooth regions (for example, natural tooth regions) in each of a plurality of images, compare the proportion of tooth regions (for example, the number of pixels constituting teeth), and select the image with the highest proportion or the top predetermined number as the evaluation image, or it may select an image in which the proportion is equal to or greater than a predetermined threshold as the evaluation image. It can also be said that the second image generation unit 75 acquires an image as the evaluation image in which the proportion of natural tooth regions within the field of view is equal to or greater than a predetermined threshold from among a plurality of images in which predetermined image processing has been performed.

[0089] This allows for plaque detection using images that capture a larger portion of the tooth area, thereby improving the accuracy of plaque detection and providing users with more useful information. The proportion of the tooth area is calculated by dividing the area of ​​the tooth area in the image (e.g., number of pixels) by the total area of ​​the image (e.g., number of pixels), but is not limited to this method.

[0090] Next, the second image generation unit 75 detects tooth regions in the evaluation image (S106). For example, the second image generation unit 75 detects tooth regions in the evaluation image using the same method as the tooth region detection process by the first image generation unit 72 described above. If tooth regions have been detected by the first image generation unit 72, the second image generation unit 75 may use the results. Alternatively, the first image generation unit 72 may perform a simplified (low-accuracy) detection of tooth regions, and the second image generation unit 75 may perform a more detailed (high-accuracy) detection of tooth regions again.

[0091] For example, the second image generation unit 75 generates an HSV image by converting the color space of the evaluation image (e.g., an RGB image) to the HSV space.

[0092] Next, the second image generation unit 75 detects plaque in the tooth region (S107). The second image generation unit 75 detects a region in the HSV image where at least one of the multiple pixels in the tooth region satisfies the following conditions: saturation is within a first predetermined range (for example, 30 to 80 in 8-bit representation), hue is within a second predetermined range (for example, 140 to 170 in 8-bit representation), and brightness is within a third predetermined range (for example, 100 to 180 in 8-bit representation). The first predetermined range, the second predetermined range, and the third predetermined range are not limited to the above numerical ranges, as long as they distinguish between the actual plaque region and other regions.

[0093] The second image generation unit 75 may use an HSL image instead of an HSV image. In this case, the second image generation unit 75 generates an HSL image by converting the color space of the evaluation image (RGB image) to the HSL space. The second image generation unit 75 then detects a region in the HSL image containing multiple pixels in the tooth region where at least one of the following is located: saturation within a fourth predetermined range, hue within a fifth predetermined range, and luminance within a sixth predetermined range.

[0094] Next, the second image generation unit 75 generates a plaque-enhanced image that displays the tooth region detected in the evaluation image (S108). The plaque-enhanced image may be, for example, an image in which the plaque region is superimposed on the evaluation image. In this way, the second image generation unit 75 detects the tooth region in the evaluation image included in a plurality of images that have undergone predetermined image processing, and generates a plaque-enhanced image that shows the presence or absence of plaque in the detected tooth region (tooth). The tooth region includes, for example, the natural tooth region.

[0095] Next, the second image generation unit 75 generates and outputs notification information to inform the user that a plaque-enhanced image has been generated (S109). For example, if the determination unit 74 determines that the toothbrush has stopped and the generation of the plaque-enhanced image is complete, the second image generation unit 75 may output a notification image as notification information to inform the user that a plaque-enhanced image has been generated. For example, the notification information is a notification image that is notified by image, but it may also be information that is notified by stimuli such as sound, light, or vibration.

[0096] Figure 11 shows an example of a notification image P2 according to this embodiment. The notification image P2 is a display image used to notify that a plaque-enhanced image has been generated. The notification image P2 is displayed by the display unit 76.

[0097] As shown in Figure 11, the notification image P2 is an image in which notification information 203, which notifies that a plaque-enhanced image has been generated, is superimposed on the currently displayed live image P1. For example, the notification information 203 may include information that allows the user to choose whether or not to display the plaque-enhanced image. The notification information 203 may also include information that indicates that the image will automatically switch to the plaque-enhanced image (for example, information that indicates that the image will switch to the plaque-enhanced image after A seconds), or it may include information that notifies the user that the creation of the plaque-enhanced image is complete.

[0098] Furthermore, the display manner of the notification information 203 may differ depending on the area of ​​the plaque region in the plaque-enhanced image. The notification information 203 may be displayed with greater emphasis the larger the area of ​​the plaque region is above a threshold. For example, the notification information 203 may be displayed in larger characters the larger the area of ​​the plaque region is.

[0099] Furthermore, the notification information 203 is not limited to textual information; it may also be a designated mark, an image of a reduced-size plaque-enhancing image, or an image of a cropped portion of a plaque-enhancing image.

[0100] Referring again to Figure 9, the determination unit 74 determines whether or not it has detected a predetermined operation indicating a switch to a plaque-enhanced image (S110). The determination unit 74 makes the determination in step S110, for example, by determining whether or not the receiving unit 73 has received a predetermined operation indicating a switch to a plaque-enhanced image.

[0101] Next, if the determination unit 74 determines that it should switch to a plaque-enhanced image, the second image generation unit 75 displays the plaque in the tooth region by displaying the plaque-enhanced image (S111). Displaying the plaque in the tooth region is one example of outputting a plaque-enhanced image.

[0102] Figure 12 shows an example of a plaque-enhanced image P3 according to this embodiment. The plaque-enhanced image P3 can also be described as a plaque region image showing the plaque area. The plaque-enhanced image P3 is displayed by the display unit 76.

[0103] As shown in Figure 12, the plaque-enhanced image P3 is an image in which the area in the tooth region where the concentration of fluorescent substance is above a predetermined value (plaque region 202) is superimposed on the evaluation image. The display unit 76 may also superimpose and display the area in the tooth region 201 where plaque is present (in this case, the plaque region 202) onto the evaluation image. The plaque region 202 may be highlighted, for example.

[0104] Note that in Figure 12, the image includes the toothbrush portion 11a in addition to the tooth region 201, but the toothbrush portion 11a does not necessarily have to be included.

[0105] Figure 12 shows an example where information indicating the presence or absence of plaque (binary) is detected as a plaque region, but the amount of plaque (multi-level) may also be detected. For example, the accumulation level (concentration or density) of fluorescent material can be detected by comparing the intensity of red fluorescence per unit area (fluorescence intensity) of the plaque region. For example, the second image generation unit 75 may detect the accumulation level based on the brightness V value of the HSV image of the plaque region. Alternatively, the second image generation unit 75 may detect the accumulation level based on the brightness L value of the HSL image of the plaque region. Furthermore, in the displayed image, the display mode may be varied according to the accumulation level. For example, the second image generation unit 75 may display the plaque in the region where plaque exists in a different manner according to the accumulation level. Varying the display mode means, for example, performing grayscale display or color display according to the accumulation level, but is not limited to these. The grayscale display may be, for example, a display of three or more grayscale levels (i.e., three or more accumulation levels). Furthermore, the color display may consist of, for example, three or more colors (i.e., three or more levels of accumulation). In this way, the toothbrush system can determine the extent of plaque accumulation on the teeth by using the intensity of red fluorescence, and display the plaque area in a display manner corresponding to the accumulation level. The accumulation level can also be referred to as the deposition level.

[0106] Referring again to Figure 9, the determination unit 74 then determines whether a predetermined period of time has elapsed since switching from the live image P1 to the plaque-enhanced image P3 (S112). The predetermined period is set in advance and stored in the memory unit (not shown).

[0107] If the determination unit 74 determines that a predetermined period has elapsed (Yes in S112), it resumes displaying the live image P1 (S113). In other words, when a predetermined period has elapsed, the determination unit 74 automatically switches from the plaque-enhanced image P3 to the live image P1. If the determination unit 74 determines that a predetermined period has not elapsed (No in S112), it displays the plaque-enhanced image P3 until the predetermined period has elapsed.

[0108] Furthermore, if the determination unit 74 determines that it is not necessary to switch to the plaque-enhanced image P3 (No in S110), the process is terminated. In other words, the live image P1 continues to be displayed.

[0109] Thus, the display unit 76 may, for example, display the plaque-enhanced image P3 during the period when the determination unit 74 determines it to be in a stationary state, and output the live image P1 (for example, a video containing multiple second RGB images) during other periods. Alternatively, if the plaque-enhanced image P3 is selected (it is selected to display it), the display unit 76 may display the plaque area image for a predetermined period, and after the predetermined period has elapsed, switch to a video containing multiple second RGB images and output the video.

[0110] In step S103, an example was described in which a plaque-enhanced image P3 is generated when a predetermined user operation is detected. However, the trigger for generating the plaque-enhanced image P3 is not limited to a predetermined user operation. For example, the plaque-enhanced image P3 may be automatically generated at a predetermined time after brushing has started (an example of a predetermined timing), or it may be generated by other triggers. The predetermined period may be, for example, 30 seconds, 1 minute, less than 30 seconds, or longer than 1 minute.

[0111] Although the example described uses still images for the plaque area, there may be one or more still images. Furthermore, the plaque area images may also be videos.

[0112] Note that notification information does not necessarily need to be generated. For example, when plaque-enhanced image P3 is generated, plaque-enhanced image P3 may be automatically displayed.

[0113] In step S112, the display of the live image P1 may be resumed, for example, when a predetermined operation is performed by the user. In other words, the display of the live image P1 may be resumed without the predetermined period of time having elapsed.

[0114] In this example, plaque detection and display of plaque areas are described only for the plaque area. However, similar plaque detection and display of plaque areas may also be performed for the brush area, including the brush portion of the toothbrush part 11a or 11b, in addition to the plaque area. Alternatively, similar plaque detection and display of plaque areas may be performed for all areas of the evaluation image.

[0115] Furthermore, from the perspective of prioritizing real-time performance, the time from when "Yes" is determined in step S103 until the plaque-enhanced image P3 is displayed should be as short as possible. For example, this time may be 2 minutes or less, 1 minute or less, or 30 seconds or less.

[0116] (Other Embodiments) Although a toothbrush system according to the embodiments of this disclosure has been described above, this disclosure is not limited to these embodiments.

[0117] For example, the above description mainly described the processing performed on the mobile terminal 70. At least some of these processes may be performed on the main unit 10 or the cloud server 80.

[0118] Furthermore, while the above example uses plaque in the tooth area as an example of oral cavity conditions, the toothbrush system is not limited to this. It may also determine the degree of gum swelling, periodontal pocket depth, oral cavity contamination, and gum injuries based on evaluation images, and display the determination results superimposed on the evaluation images during brushing. This allows users to understand areas in the oral cavity where they should focus their brushing efforts and areas where they should not apply too much pressure, thus supporting the user in brushing effectively.

[0119] Furthermore, in the above embodiment, whether or not brushing is in progress can be detected, for example, by changes in the position or orientation of the toothbrush parts 11a and 11b, or by the shape of the bristles of the toothbrush parts 11a and 11b as seen in the image.

[0120] Furthermore, each processing unit included in the toothbrush system according to the above embodiment is typically implemented as an LSI, which is an integrated circuit. These may be individually integrated into a single chip, or some or all of them may be integrated into a single chip.

[0121] Furthermore, integrated circuit implementation is not limited to LSIs; it may also be achieved using dedicated circuits or general-purpose processors. Alternatively, an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor capable of reconfiguring the connections and settings of circuit cells within the LSI, may be used.

[0122] Furthermore, in each of the above embodiments, each component may be implemented by being composed of dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0123] Furthermore, this disclosure may be implemented as a control method or the like performed by a toothbrush system. Alternatively, this disclosure may be implemented as a main unit, a portable terminal, or a cloud server included in a toothbrush system.

[0124] Furthermore, the division of functional blocks in the block diagram is just one example; multiple functional blocks can be implemented as a single functional block, a single functional block can be divided into multiple parts, or some functions can be moved to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can be processed in parallel or time-sharing by a single piece of hardware or software.

[0125] Furthermore, the order in which each step in the flowchart is performed is illustrative for the purpose of specifically illustrating this disclosure, and may be in a different order. Also, some of the above steps may be performed simultaneously (in parallel) with other steps. Also, some of the above steps may not be performed.

[0126] The toothbrush system, etc., relating to one or more embodiments has been described above based on embodiments, but this disclosure is not limited to these embodiments. Without departing from the spirit of this disclosure, various modifications that a person skilled in the art can conceive of may be applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments.

[0127] This disclosure is applicable to toothbrush systems.

[0128] 10 Main body 10a Head 10b Handle 10b1 Operation button 10c Neck 11a, 11b Toothbrush 11c Light-shielding member 12 Imaging optical system 14 Image sensor 26A, 26B, 26C, 26D LED 36, 40 Actuator 50 Central control unit 54 LED control unit 56 Lens driver 58 Wireless communication module 60 Power control unit 62 Controller 64 Memory 66 Battery 68 Coil 69 Charger 70 Mobile terminal 71 Acquisition unit 72 First image generation unit 73 Reception unit 74 Judgment unit 75 Second image generation unit 76 Display unit (output unit) 80 Cloud server 90 Position sensor 201 Tooth area 202 Plaque area 203 Notification information P1 Live image P2 Notification image P3 Plaque-enhanced image U User

Claims

1. A toothbrush system comprising: a toothbrush for oral care; a light source that emits light of a predetermined wavelength that plaque reacts to onto a care area including the tooth region; an imaging unit that acquires a plurality of first RGB images including the care area and reflected and fluorescent light from the care area while brushing with the toothbrush; a first image generation unit that generates a plurality of second RGB images by performing image processing on the plurality of first RGB images to make the teeth grayscale; a second image generation unit that detects the presence or absence of plaque in the tooth region based on evaluation images included in the plurality of second RGB images at a timing corresponding to the user's operation during brushing, and generates a plaque-enhanced image indicating the presence or absence of plaque in the tooth region; and an output unit that outputs the plaque-enhanced image.

2. The toothbrush system according to claim 1, comprising a determination unit for determining the stationary state of the toothbrush, wherein the user's operation includes the user stopping the toothbrush during brushing, and the timing includes the timing at which the determination unit determines that the toothbrush has stopped during brushing.

3. The toothbrush system according to claim 2, wherein the output unit outputs a notification image to notify that the plaque-enhanced image has been generated when the determination unit determines that the toothbrush has come to a stop during brushing and the generation of the plaque-enhanced image has been completed.

4. The toothbrush system according to claim 2, wherein the output unit outputs the plaque-enhancing image during the period that the determination unit determines to be the stationary state, and outputs a video including the plurality of second RGB images during the other period during brushing.

5. The toothbrush system according to claim 2, wherein the determination unit determines that the toothbrush is in a stationary state if, in two or more consecutive second RGB images among the plurality of second RGB images, the movement of the toothbrush region within the image is less than a predetermined first threshold.

6. The toothbrush system according to any one of claims 1 to 5, wherein the user operation includes an operation to display the plaque-enhanced image generated by the second image generation unit, and when the operation is detected, the output unit outputs the plaque-enhanced image for a predetermined period of time, and after the predetermined period has elapsed, switches to a video including the plurality of second RGB images and outputs the video.

7. The toothbrush system according to any one of claims 1 to 5, wherein the second image generation unit acquires a second RGB image from among the plurality of second RGB images in which the proportion of pixels constituting natural teeth within the field of view is equal to or greater than a second threshold, as the evaluation image.

8. A control method for a toothbrush system, the toothbrush system comprising: a toothbrush for caring for the oral cavity; a light source that emits light of a predetermined wavelength to which plaque reacts toward a care area including a tooth region; and an imaging unit that acquires a plurality of first RGB images including the care area and reflected light and fluorescence from the care area while brushing with the toothbrush, the control method comprising: generating a plurality of second RGB images by performing image processing on the plurality of first RGB images to make the teeth grayscale; detecting the presence or absence of plaque in the tooth region based on evaluation images included in the plurality of second RGB images at a timing corresponding to the user's operation during brushing; generating a plaque-enhanced image showing the presence or absence of plaque in the tooth region; and outputting the plaque-enhanced image.

9. A program that causes a computer to execute the control method described in claim 8.

Images