Tracking cleaned areas

Abstract

A system for tracking areas which have been cleaned using a cleaning appliance is described, The system comprises a location determination module; and an audio detection module configured to detect sound from its surroundings. The audio detection module is configured to detect when a cleaning appliance is operating based on the detected sound. The location determination module is configured to, in response to the audio detection module detecting that the cleaning appliance is operating, determine location information indicative of a location of a cleaning portion of the cleaning appliance. A method for tracking areas which have been cleaned using a cleaning appliance is also described.

Description

[0001] 1 P004482-W001

[0002] TRACKING CLEANED AREAS

[0003] BACKGROUND

[0004] When a user is cleaning an area, e.g., with a vacuum cleaner, it is often difficult to keep track of which areas have already been cleaned, and which are yet to be cleaned during the cleaning process. Dust particles can be very small such that it is difficult, if not impossible, to see them with the naked eye. It can therefore be difficult to distinguish which areas have already been cleaned. Although vacuum cleaners can leave stripes in long-pile carpets that indicate where the vacuum cleaner has been, this can be unreliable, and there is no such indicator of the previous path of a cleaning appliance when cleaning areas with hard or smooth surfaces, for example wooden floors.

[0005] This difficulty in keeping track of the areas that have already been cleaned can often lead to a duplication of effort on the user’s part and inefficiencies in automated systems.

[0006] SUMMARY

[0007] The present invention relates to systems and methods for tracking areas which have been cleaned using a cleaning appliance.

[0008] In particular, according to a first aspect, there is provided a system for tracking areas which have been cleaned using a cleaning appliance, the system comprising: a location determination module; and an audio detection module configured to detect sound from its surroundings, wherein: the audio detection module is configured to detect when a cleaning appliance is operating based on the detected sound; and the location determination module is configured to, in response to the audio detection module detecting that the cleaning appliance is operating, determine location information indicative of a location of a cleaning portion of the cleaning appliance.

[0009] By determining the location information in response to the detection of a sound indicating that the cleaning appliance is operating, there is no requirement to communicatively couple 2 P004482-W001

[0010] the cleaning appliance with the system. The system can therefore be used with one or more different cleaning appliances, and there is no need to add connectivity systems to the cleaning appliances to allow them to communicate with the system for tracking the areas that have been cleaned.

[0011] By determining the location information indicative of a location of the cleaning portion of the cleaning appliance, areas which have been cleaned can be tracked, because the cleaning portion may define an area being cleaned by the cleaning appliance at any given time.

[0012] Optional features will now be set out. These optional features are applicable singly or in combination with any aspect.

[0013] The system may only begin determining the location information after the sound of the cleaning appliance operating has been detected. This provides a more efficient system as the location information is only determined whilst the cleaning appliance is in operation. Furthermore, this automation also means that the user is not required to manually instruct the location determination module to start determining location information, therefore providing ease of use for the user.

[0014] The cleaning appliance may be configured to clean a surface. The cleaning appliance may be configured to clean the surface under a user’s direction, for example, by cleaning the surface as the user manually moves the appliance across the surface. Alternatively, or in addition, the cleaning appliance may be configured to clean the surface in an automated manner. For example, the cleaning appliance may be configured to move across and clean a surface under its own direction by following an automation protocol. In a specific example, the cleaning appliance may be a robotic vacuum cleaner configured to move across a floor surface. Such a robotic vacuum cleaner may be configured to clean dust particles from said surface as part of an automated cleaning routine, which may have been previously defined by a user and / or automatically based on an environment in which the surface is located. 3 P004482-W001

[0015] The cleaning appliance may be a floor cleaning appliance, e.g., configured to clean a (surface of a) floor. The cleaning appliance may be configured to clean a surface, such as a floor surface, by removing dust and / or dirt from the surface as the cleaning appliance (and in particular, the cleaning portion of the cleaning appliance) moves across the surface. The cleaning portion of the cleaning appliance may remove dust and / or dirt from the surface as the cleaning appliance moves across the surface. The region of the surface from which dust particles are removed at any given time may be the region of the surface facing or engaged with the cleaning portion of the cleaning appliance during the cleaning operation. Where the surface is a floor, the region of the floor from which dust particles are removed at any given time may be the region of the floor beneath at least a part of the cleaning portion of the cleaning appliance. As such, the cleaning portion of the cleaning appliance may face or engage the surface during the cleaning operation.

[0016] The cleaning appliance may be a vacuum cleaner. During operation of a vacuum cleaner, (e.g., during “vacuuming”), dust particles on the surface (e.g., floor) may be entrained in an airflow which causes them to be removed from the floor by suction and deposited in a receptacle which forms part of the vacuum cleaner.

[0017] For completeness, the cleaning appliance may be any appliance suitable for cleaning, vacuuming, dusting, wiping, steaming and / or treating a surface, such as a vacuum cleaner, a wet vacuum cleaner, a dry and wet vacuum cleaner, a polisher, a steam cleaner, a hard surface cleaner or a carpet cleaner, for example.

[0018] As set out above, the location determination module is configured to determine location information indicative of a location of the cleaning portion of the cleaning appliance in response to the audio detection module detecting that the cleaning appliance is operating. The location determining module may be configured to determine location information indicative of a location of the cleaning portion of the cleaning appliance at any given time when the audio detection module detects that the cleaning appliance is operating. As such, the location determining module may determine a track of the location of the cleaning portion of the cleaning appliance over time as the cleaning portion of the cleaning appliance moves. 4 P004482-W001

[0019] The location determination module may be configured to determine the location information only when the audio detection module detects that the cleaning appliance is operating based on the detected sound. As such, the location determination module is configured to stop detecting location information in response to the audio detection module detecting that the cleaning appliance has stopped operating.

[0020] The location determination module may be configured to determine the location information at predetermined time intervals (during the detected operation of the cleaning appliance). The time intervals may be regular time intervals, such that each time interval has the same duration. The duration of the time intervals may be configured to provide approximately continuous location monitoring for the cleaning portion of the cleaning appliance. For example, by reducing the duration of the time intervals to an appropriate amount of time, the location information may be a continuous, or near-continuous, set of time series data.

[0021] The system may further comprise a processor. The processor may be configured to generate instructions that cause a display component to display a visual representation of the location information. The instructions may cause the display component to display the visual representation of the location information in real-time (e.g., such that the visual representation of the location information is displayed at the same time or substantially the same time as the cleaning appliance is at the displayed location. As such, the visual representation of the location information may be displayed at the display component during the cleaning operation, whilst the cleaning portion of the cleaning appliance is at the relevant location. This may provide substantially immediate feedback to a user about the location of the cleaning portion of the cleaning appliance as the cleaning appliance moves. As the cleaning appliance is moved (e.g., across a surface), the location determination module may update the location information indicative of a location of the cleaning portion of the cleaning appliance, and thus update the instructions generated by the processor to cause the display to show a visual representation of the location information as it is being updated. 5 P004482-W001

[0022] The audio detection module may be configured to detect when a cleaning appliance is operating based on the detected sound using a machine learning model. In particular, the audio detection module may be configured to detect when a cleaning appliance is operating by inputting data corresponding to the detected sound into a machine learning model trained to detect when a cleaning appliance is operating; and receiving, from the trained machine learning model, an output indicating that the cleaning appliance is operating. The machine learning model may comprise one or more of the following: convolutional neural networks (CNNs), which may be ID and / or 2D CNNs, vanilla neural networks, Long Short-term Memory (LSTM) neural networks, Correlation Recurrent Units (CRUs), Recurrent Neural Networks (RNNs), tree-based methods, and histogram-based methods. The audio detection module may be configured to detect when a cleaning appliance is operating by thresholding the sound signal after a band-pass filter centred on a motorfrequency.

[0023] The machine learning model may be trained using a plurality of captured sound recordings. The plurality of sound recordings may be captured by a mobile device, for example.

[0024] The captured sound recordings may include captured sound recordings of one or more cleaning appliances during operation (e.g., cleaning of a surface). In this way, the machine learning model can be trained to detect when a cleaning appliance is operating based on the sound detected by the audio detection model.

[0025] The audio detection module may be configured to distinguish the sound of a cleaning appliance operating from one or more different background sounds. In this way, the system can avoid accidental triggering of the location determination module by other background sounds, such as a kitchen blender, for example. In other words, not just any sound will trigger the location determination module to determine the location information. Instead, the machine learning model is trained to recognise the sound of an operating cleaning appliance compared to other sounds, and only trigger the location determination module to determine the location information when the sound of the cleaning appliance operating is recognised. 6 P004482-W001

[0026] The machine learning model may be trained using captured sound recordings of one or more different background sounds. Background sounds may include one or more of, for example, a kitchen blender operating, a dishwasher operating, a human talking, a washing machine operating, a tumble drier operating, a kitchen mixer operating, a radio or other music player playing, a hedge trimmer operating, a lawnmower operating, any other sound other than a cleaning appliance operating. In this way, the machine learning model may be trained to recognise and distinguish the sound of an operating cleaning appliance from other background sounds, which may be similar.

[0027] The audio detection module may be configured to distinguish the sound of a particular cleaning appliance operating from one or more different background sounds, and / or from one or more other different cleaning appliances. In this way, the location determination module may only be triggered when that particular cleaning appliance is operating. The machine learning model may be able to distinguish the sound of a particular cleaning appliance, because it is trained using captured sound recordings of the particular cleaning appliance operating. As set out above, the machine learning model may also be trained using captured sound recordings of one or more different background sounds. The machine learning model may additionally / alternatively be trained using captured sound recordings of one or more other different cleaning appliances. In this way, the trained machine learning model can be trained to distinguish the sound of the particular cleaning appliance operating from other different cleaning appliances and / or background noise. As such, the location determination module may be triggered to determine the location information only when the particular cleaning appliance is operating.

[0028] The particular cleaning appliance may be a particular type of cleaning appliance (e.g., a robotic vacuum cleaner rather than a manually operated (e.g., hand operated) vacuum cleaner). As such, the machine learning model may be trained to detect that a particular type of cleaning appliance is operating based on the detected sound. The machine learning model may be trained using captured sound recordings of the particular type of cleaning appliance operating (e.g., a robotic vacuum cleaner operating). 7 P004482-W001

[0029] The machine learning model may be trained to detect a type of cleaning appliance operating based on the detected sound. The machine learning model may be trained using captured sound recordings of a plurality of different types of cleaning appliance operating. In this way, the machine learning model can be trained to recognise the respective sounds of the different types of cleaning appliance and to identify the type of cleaning appliance operating.

[0030] The particular cleaning appliance may be a particular brand and / or model of cleaning appliance. As such, the machine learning model may be trained to detect that a particular brand and / or model of cleaning appliance is operating based on the detected sound. The machine learning model may be trained using captured sound recordings of the cleaning appliance of that particular brand and / or model operating.

[0031] The machine learning model may be trained to detect a brand and / or model of the cleaning appliance operating based on the detected sound. The machine learning model may be trained using captured sound recordings of the operation of a plurality of different brands and / or models of cleaning appliances. The machine learning model can then be trained to recognise the respective sounds of these different brands and / or models operating and identify the brand and / or model of the cleaning appliance from the sound.

[0032] The audio detection module may be configured to detect when a cleaning appliance is cleaning a surface based on the detected sound. For example, the audio detection module may be able to distinguish when the cleaning portion is cleaning a surface (e.g., when the cleaning portion is in contact with a surface or is facing the surface, and is powered on). In this way, the location determination module may only be triggered when the cleaning appliance is cleaning a surface (e.g., such that the cleaning portion is in contact with the surface), rather than when it is simply powered on but the cleaning portion is not in contact with the surface.

[0033] The audio detection module may be configured to (and in particular the machine learning model may be trained to) detect a type of surface being cleaned by the cleaning appliance during operation based on the detected sound. The machine learning model may be trained 8 P004482-W001

[0034] using captured sound recordings of one or more cleaning appliances operating on different types of surface. Example types of surface may include a wood surface, a carpet surface, a tiled surface, a laminate surface, a vinyl surface, a stone surface, a linoleum surface, a concrete surface, a parquet surface, for example. The machine learning model can then be trained to recognise the respective sounds of one or more cleaning appliances cleaning different types of surface, and identify a type of surface being cleaned by the cleaning appliance based on the sound.

[0035] The instructions generated by the processor may then cause the display component to display a visual representation of the type, brand and / or model of cleaning appliance operating, and / or the type of surface being cleaned by the cleaning appliance.

[0036] We now discuss ways in which the location information may be determined by the location determination module.

[0037] The system may further comprise an environment sensing system configured to capture mapping data of its surroundings. The location determination module may be configured to determine the location information based on the captured mapping data. In particular, by capturing mapping data of the surroundings of the environment sensing system, the location determination module can determine location information indicative of a location of a cleaning portion of the cleaning appliance in the surroundings.

[0038] The environment sensing system may comprise one or more of: a camera configured to capture video data of its surroundings; a light detection and ranging (LiDAR) unit configured to capture mapping data of its surroundings; and an inertial measurement unit (IMU) configured to detect changes in motion and orientation of the cleaning appliance and to generate IMU data in response to the detection.

[0039] When the environment sensing system comprises a camera configured to capture video data of its surroundings, the location determination module may be configured to determine the location information based on at least the captured video data. The term “video data” may comprise a time series of individual frames captured by the camera at 9 P004482-W001

[0040] each time interval or a continuous stream of image frames. In other words, the video data may comprise a series of individual images captured by the camera. The camera may be adapted to capture a single image at every time interval. Alternatively, the camera may capture a continuous stream of images, but only single frames are selected for transfer to the location determination module at each time interval. The camera may be configured to operate in the visible spectrum of light or in any other appropriate spectrum of light, such as infrared. The camera may be configured to capture the video data at predetermined time intervals. The time intervals are preferably regular time intervals, meaning that each time interval is the same duration. The duration of the time intervals may be configured to provide more or less continuous location monitoring for the cleaning appliance. For example, by reducing the duration of the time intervals to an appropriate amount of time, the video data may be a continuous, or near continuous, set of time series data.

[0041] By way of a worked example of the system in use, the user may begin cleaning a surface by positioning the cleaning appliance at a first location on the surface at a first point in time. In response to the audio detection module detecting that the cleaning appliance is operating, the camera may be configured to capture video data from the first location and the location determination module determines location information indicative of the first location on the surface where the cleaning appliance is located at that first point in time based on the captured video data. The user may then move the cleaning appliance across the surface to clean the surface. As the user move the cleaning appliance, the cleaning appliance will arrive at a second location on the surface at a second point in time, the first and second points in time being separated by a single time interval. At the second point in time, or after a time interval has elapsed, the camera may be configured to capture further video data and the location determination module determines location information indicative of the second location on the surface where the cleaning appliance is located at that second point in time based on the further video data. This process may be repeated across a plurality of locations and time intervals as the surface is being cleaned, thereby generating a series of individual location data signals illustrating the changing position of the cleaning appliance (and in particular the cleaning portion of the cleaning appliance) on the surface over time. 10 P004482-W001

[0042] Preferably, the display component is configured to display the captured video data and the instructions are configured to cause the display component to superimpose the visual representation of the location information over the display of the captured video data, such that each locus of the visual representation of the location information is superimposed on the corresponding locus of the captured video data. The term “corresponding locus” may be interpreted as meaning that a given point in real space (i.e., the surroundings of the cleaning appliance) has the same locus position in both the captured video data and the visual representation of the location information. Put another way, a locus in the captured video data may correspond to a locus in the visual representation of the location information, and both loci may correspond to the same point in real space.

[0043] The visual representation of the location data may take different forms according to various aspects of the invention. For example, the visual representation of the location data may include a coordinate, or a distance moved from an origin point, which may be defined at the point of activation of the cleaning appliance for cleaning the surface. Alternatively, the user may define, or redefine, the origin point manually. For example, the user may define the origin point when they begin to clean a first surface with the cleaning appliance and then redefine the origin point when they begin to clean a second, different, surface. In a practical example, this may occur as the user moves from room to room.

[0044] In a further example, the visual representation of the location data may include one or more colours for representing the location of the cleaning apparatus on the surface. In this case, a parameter of the colour may change as the location of the cleaning apparatus moves across the surface being cleaned. The parameter of the colour may comprise one or more of: opacity; hue; saturation; brightness; and the like. For example, where the visual representation of the location data includes a single colour, the colour may become more opaque, more saturated and / or brighter as the cleaning appliance passes over a given position on the surface being cleaned one or more times, meaning that an opaque, saturated and / or bright colour on the display may be indicative of a clean area of the surface. Alternatively, again where the visual representation of the location data includes a single colour, the colour may become less opaque, less saturated and / or darker as the cleaning appliance passes over a given position on the surface being cleaned one or more times, 11 P004482-W001

[0045] meaning that a translucent, unsaturated and / or dark colour on the display may be indicative of a clean area of the surface. In a further example, where the visual representation of the location data includes multiple colours, the hue of the colour may change as the cleaning appliance passes over a given position on the surface being cleaned one or more times, meaning that a given colour, which may be predetermined or set by the user, may be indicative of a clean area of the surface.

[0046] In a further example, the visual representation of the location data may include a visual pattern for representing the location of the cleaning apparatus on the surface. In this case, a parameter of the pattern may change as the location of the cleaning apparatus moves across the surface being cleaned. The parameter of the pattern may comprise one or more of: an opacity; a size; an orientation; a repetition of the pattern; a colour (which may include the sub-parameters of hue, brightness and saturation discussed above); and the like. For example, when the user initiates the cleaning appliance, the entire surface may be assigned a given pattern and as the cleaning appliance moves across the surface to clean the surface, the pattern may change or be removed. In a specific example, the visual representation of the location data may show a pattern fading (i.e., reducing in opacity) as the cleaning device moves over a given location one or more times.

[0047] When the environment sensing system comprises a light detection and ranging (LiDAR) unit configured to capture mapping data of its surrounding, the location determination module may be configured to determine the location information based on at least the captured mapping data.

[0048] A LiDAR unit may be understood as a component for determining ranges, or distances, between the unit and another object. Such a LiDAR unit functions by generating a beam of light, preferably using a laser, and targeting the light at an object or a surface. The LiDAR unit is adapted to detect the light reflected by the object and to measure the time taken for the light to travel from the LiDAR unit to the object and back, which may be referred to as a time-of-flight measurement. Based on the time-of- flight measurement and the known speed of light, the LiDAR unit may determine a distance, or range, between the LiDAR unit and an object or surface reflecting the emitted light. By performing a number of 12 P004482-W001

[0049] measurements across the environment, for example by scanning the laser over the surroundings of the LiDAR unit, it is possible to map the layout of an area. The accuracy of this mapping may be improved further by repeating the above process with the LiDAR unit from multiple locations within the environment.

[0050] A LiDAR unit may include one or more of the following components. The LiDAR unit may comprise a laser, a sensor and an actuated mirror. The lasers may be configured to emit light in a wavelength range of 500nm to 1600nm, and preferably in the range of 600nm to lOOOnm. The laser may be power-limited in order to render the laser eye-safe for the user. The laser may be operated in a pulsed manner or a continuous manner according to the application of the cleaning appliance. Preferably, the laser is a 600nm to lOOOnm laser operated in a pulsed manner. The pulse frequency may be configured to provide more or less continuous location monitoring for the cleaning appliance. For example, by increasing the pulse frequency to an appropriate frequency the captured mapping data, and so the location information, may be a continuous, or near continuous, set of time series data.

[0051] The location determination module may utilize the mapping data from the LiDAR unit in a corresponding way to the captured video data discussed above in relation to the camera.

[0052] The environment sensing system may comprise both a LiDAR unit and a camera as described above. In these examples, the mapping data captured by the LiDAR unit, and the video data captured by the camera may be utilized in conjunction with each other by the location determination module to determine the location information. In this way, the accuracy of the location information can be improved. In particular, the captured mapping data may be used to provide distance measurements to objects or surfaces identified in the captured video data, thereby improving the accuracy of the determined location information and so improving the accuracy of tracking the areas cleaned by the cleaning appliance.

[0053] When the environment sensing system comprises an inertial measurement unit (IMU) configured to detect changes in motion and orientation of the cleaning appliance and to 13 P004482-W001

[0054] generate IMU data in response to the detection, the location determination module may be configured to determine the location information based on at least the IMU data.

[0055] The IMU may comprise one or more components for detecting changes in motion and orientation of the cleaning appliance. For example, the IMU may comprise an accelerometer, which may be used to detect an acceleration of the cleaning appliance in a given direction. Further, the accelerometer may be configured to determine the orientation of the cleaning appliance by measuring the action of gravity on the accelerometer. Alternatively, or in addition, the IMU may comprise a gyroscope sensor. The gyroscope sensor may be adapted to measure the orientation of the cleaning appliance. The orientation of the cleaning appliance may be measured in any number of directions or planes. For example, the orientation may be measured in a Cartesian coordinate system having three orthogonal axes. Alternatively, the orientation may be measured as pitch, yaw and roll, which may be defined in relation to the position the cleaning appliance was in when first activated or in relation to a predefined reference position. The IMU may contain any number of accelerometers and / or gyroscopes. For example, the IMU may contain a single accelerometer or gyroscope, an accelerometer and a gyroscope, three accelerometers (each arranged to measure acceleration of the cleaning appliance in one of three orthogonal directions) and the like.

[0056] The IMU data may be used to determine the location information as follows. When the cleaning appliance is first initiated, the IMU unit may be activated and a reference set of IMU data taken when the cleaning appliance is at rest. When the user moves the cleaning appliance to begin the cleaning process, the IMU unit will detect the acceleration of the cleaning appliance in a given direction. Based on the acceleration signals and the time between changes in acceleration, the IMU unit, or the location determination module, may determine the distance moved by the cleaning appliance across the surface being cleaning. Accordingly, the location determination module may track the movements of the cleaning appliance over the surface being cleaned.

[0057] The IMU unit may be configured to capture the location IMU data at predetermined time intervals. The time intervals are preferably regular time intervals, meaning that each time 14 P004482-W001

[0058] interval is the same duration. The duration of the time intervals may be configured to provide more or less continuous location monitoring for the cleaning appliance. For example, by reducing the duration of the time intervals to an appropriate amount of time, the IMU data may be a continuous, or near continuous, set of time series data.

[0059] The location determination module may utilize the IMU data in a corresponding way to the captured video data discussed above in relation to the camera.

[0060] The IMU unit described above may be used in conjunction with the camera described above and / or the LiDAR unit described above. For example, the captured IMU data and the LiDAR captured mapping data and / or the captured video data may be utilized in conjunction with each other by the location determination module to determine the location information. In particular, the captured IMU data may be used to provide motion and / or orientation information of the cleaning portion of the cleaning appliance, which may be used to inform how the perspective of object or surfaces in the view of the captured video data and / or captured mapping data may have changes, thereby improving the accuracy of the determined location information and so improving the accuracy of tracking the surfaces cleaned by the cleaning appliance.

[0061] As set out above, the cleaning component may define a portion of a surface being cleaned at any given time. The system (e.g., the processor and / or the location determination module) may be configured to detect when the cleaning component is in contact with the surface. For example, the location determination module may be configured to detect when the cleaning component is in contact with the surface based on the mapping data captured by the environment sensing system. Alternatively / additionally, the system may be configured to detect when the cleaning component is in contact with the surface based on measurements of one or more sensors. The one or more sensors may be positioned at the cleaning appliance, for example at the cleaning portion of the cleaning appliance. The one or more sensors may include touch sensors and / or proximity sensors, for example.

[0062] The audio detection module may then be configured to start detecting sound in response to the detection that the cleaning component is in contact with the surface. Then, only when 15 P004482-W001

[0063] the audio detection module detects that the cleaning appliance is operating based on the detected sound, the location determination module may be configured to determine the location information indicative of a location of the cleaning portion of the cleaning appliance (e.g., from the mapping data).

[0064] This further improves the efficiency of the system as the audio detection module is only turned on when needed.

[0065] In some examples, the audio detection module may be configured to start detecting sound in response to a user input (e.g., a user switching the audio detection module on). In these examples, the environment sensing system and / or the location information may only switch on in response to the audio detection module detecting that the cleaning appliance is operating. As such, the environment sensing system may only start capturing mapping data, and the location determination module may only start determining the location information, in response to the audio detection module detecting that the cleaning appliance is operating.

[0066] Again, this improves the efficiency of the system as the environment sensing system and the location determination module are only turned on when the cleaning appliance is operating.

[0067] The audio detection module may comprise one or more sound sensors, including one or more acoustic sensors, ultrasonic sensors and / or vibration sensors. In examples with a vibration sensor, the audio detect module may function as a vibration detection module. The audio detection module may comprise one or more microphones, for example.

[0068] The system may comprise a mobile device. The mobile device may comprise the location determination module and / or the audio detection module. The mobile device may comprise the processor configured to generate the instructions that cause a display component to display the visual representation of the location information. The mobile device may comprise the display component itself. The mobile device may comprise the environment sensing system. 16 P004482-W001

[0069] The mobile device may be mountable to the cleaning appliance. In this way, when the mobile device comprises the audio detection module, the audio detection module can be positioned proximate to the cleaning appliance such that it can clearly detect the sound of the cleaning appliance operating.

[0070] Providing the audio detection module at the mobile device may be particularly useful when the training data used to train the machine learning model comprises sound recordings previously captured by an audio detection module of the mobile device. This is because the training data can be captured by a similar type of audio detection module to that used to detect the sound. Accordingly, the model accuracy, model precision and / or model sensitivity / recall can be improved.

[0071] A sampling rate of the audio detection module may be approximately equal to the sampling rate of the captured sound recordings used for training the machine learning model. The term “approximately” as used herein may be understood as meaning + / - 10%, or = / - 5%, for example. The sampling rate of the audio detection module may be 22050Hz, for example. This is a suitable frequency in view of aliasing considerations, although other sampling rates could alternatively be selected. The window size (the amount of time over which a waveform is sampled expressed in samples) for the audio detection module may be 22050 samples. The time record (the amount of time over which a waveform is sampled) of the audio detection module may be Is. Again, in this way, the model accuracy, model precision and / or model sensitivity / recall can be improved. The window-size choice and overlap may be chosen based on a response time and processing power of the system, and / or on the type of sound to be analysed. It has been found that the sound of a cleaning apparatus during operation generally has little temporal variation over 1 second, and so a time record of Is is appropriate. In some examples, the time record could be less than 1 second.

[0072] Mounting the mobile device to the cleaning appliance may also means that the display component of the mobile device, and thus the visual representation of the location information, is visible to the user as they are using the cleaning appliance. 17 P004482-W001

[0073] The mobile device may comprise one or more of: a smartphone; a touchscreen device; a tablet device; a smartwatch or the like.

[0074] In some examples, the cleaning appliance itself may comprise the location determination module and / or the audio detection module. The cleaning appliance may comprise the processor configured to generate the instructions that cause a display component to display the visual representation of the location information. The cleaning appliance may comprise the display component itself. The cleaning appliance may comprise the environment sensing system. The display component may be provided on the cleaning appliance such that display component, and thus the visual represented of the location data, is visible to the user as they are using the cleaning appliance.

[0075] In some examples, the location determination module and / or the processor may be located at a remote device, such as a remote server for example. In these examples, the audio detection module, and / or the environment sensing system may still be located locally to the cleaning appliance (e.g., by being comprised in the cleaning appliance itself, and / or a mobile device mountable to the cleaning appliance), in order to detect the sound of the cleaning appliance operating and to capture mapping data of the surroundings of the cleaning appliance. The display component may be located locally to the cleaning appliance, but it could also be located at a remote device, such as a mobile device remote from the cleaning appliance. This may be particularly useful for robotic vacuum cleaners, for example.

[0076] The location determination module may be comprised in the processor, or may be a separate module.

[0077] The first aspect focuses on a system for tracking areas which have been cleaned using a cleaning appliance. A corresponding second aspect provides a method for tracking areas which have been cleaned using a cleaning appliance. In particular, according to the second aspect, there is provided a method for tracking areas which have been cleaned using a cleaning appliance, the method comprising: detecting, by an audio detection module, sound 18 P004482-W001

[0078] from the surroundings of the audio detection module; detecting, by the audio detection module, when a cleaning appliance is operating based on the detected sound; and determining, by a location determination module, and in response to the audio detection module detecting that the cleaning appliance is cleaning, location information indicative of a location of a cleaning portion of the cleaning appliance. Optional features of the first aspect set out above apply equally to the second aspect, except where clearly incompatible or where context clearly dictates otherwise.

[0079] A third aspect provides a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of the second aspect.

[0080] A fourth aspect provides a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of the second aspect.

[0081] The invention includes the combination of the aspects and preferred features described above except where such a combination is clearly impermissible or expressly avoided.

[0082] BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 is a high-level system diagram including a mobile device and a cleaning appliance.

[0084] Figure 2 shows an example of the system of Figure 1 in which the cleaning appliance is a user-operated vacuum cleaner, and the mobile device is a smartphone.

[0085] Figure 3 is a schematic diagram showing components of a mobile device.

[0086] Figure 4 is a flowchart illustrating, at a high-level, the steps which are performed by the systems disclosed herein. 19 P004482-W001

[0087] Figure 5 is a schematic illustration of an example of a visual representation that may be displayed to a user.

[0088] DETAILED DESCRIPTION

[0089] Aspects and embodiments of the present invention will now be discussed with reference to the accompanying figures. Further aspects and embodiments will be apparent to those skilled in the art.

[0090] Figure l is a high-level schematic of a system that may be used to track the areas that have been cleaned by a cleaning appliance. The system 1 includes a mobile device 10, which may be a handheld device such as a smartphone or tablet, and which has stored thereon an application configured to perform the methods disclosed herein. The system also includes a cleaning appliance 20. Cleaning appliance 20 may be used to clean a surface such as a floor, and may comprise a manually operated handheld vacuum cleaner, a robotic vacuum cleaner, a wet vacuum cleaner, a dry and wet vacuum cleaner, a polisher, a steam cleaner, a hard surface cleaner or a carpet cleaner, for example. The mobile device 10 and the cleaning appliance 20 may not be communicatively coupled to one another. Instead, as described in further detail below, the mobile device 10 can detect when the cleaning appliance is operating by detecting the sound of the cleaning appliance. As such there is no need for a wired or wireless connection between the cleaning appliance and the mobile device for the mobile device to determine when the cleaning appliance is operating. In some examples, the mobile device 10 and cleaning appliance 20 may be communicatively couples, e.g., via a wireless network such as Wi-Fi networks or cellular networks.

[0091] Figure 2 shows an example of a cleaning appliance 20 and mobile device 10 of system 1, during use. Cleaning appliance 2 has a cleaning portion 22 that defines a portion of a surface being cleaned at any given time. In the example of Figure 2, the cleaning appliance 20 is a hand-held manually operated vacuum cleaner, which the user can push across the floor using a handle, and the cleaning portion 22 cleans the floor by entraining dust and dirt in a generated airflow. The mobile device 10 is attached to the cleaning appliance 20, 20 P004482-W001

[0092] e.g. via a mount, such that the user can see the mobile device 10 whilst operating the cleaning appliance 20.

[0093] Figure 3 shows an example of a mobile device 10 which may be used to track areas that have been cleaned by cleaning appliance 20. The mobile device 10 comprises a location determination module 102, and an audio detection module 104. The audio detection module 104 is configured to detect sounds from the surroundings of the mobile device 10, and the location determination module 102 is configured to determine location information indicative of a location of the cleaning portion 22 of the cleaning appliance 20 as it moves across the surface being cleaned. The audio detection module 104 may comprise a microphone, for example. The location determination module 102 may comprise one or more processors.

[0094] The mobile device 10 may also comprise a display component 106 configured to provide feedback to the user (e.g., a display screen). This feedback may include a visual representation of the location information (e.g., a visual representation of the track of the cleaning portion as it moves across the surface). The areas that have been cleaned and / or have not been cleaned can therefore be shown to the user via the display component. Although not shown, the mobile device 110 may also comprise an input receiving component such as a touch screen, which may be integrated with the display component 106. The mobile device 10 may also comprise a processor 108. The processor may be integrated with the location determination module, and / or may be a separate component. The processor 108 is configured to generate instructions for controlling the display component 106.

[0095] The mobile device 10 may also comprise an environment sensing system 110. The environment sensing system 110 comprises one or more units, sensors or modules that capture data relating to the surroundings of the mobile device 10. Therefore, when the mobile device 10 is in proximity to the cleaning appliance 20, the environment sensing system 110 can capture data relating to the location of the cleaning appliance 20, and in particular the cleaning portion 22, relative to its surroundings. This data captured by the environment sensing system 110 can then be interpreted by the location determination 21 P004482-W001

[0096] module 102 to determine a location of the cleaning portion 22 of the cleaning appliance 20 relative to its surroundings (e.g., relative to a surface such as a floor).

[0097] The environment sensing system 110 may comprise one or more of a camera configured to capture video data of its surroundings; a light detection and ranging (LiDAR) unit configured to capture mapping data of its surroundings; and an inertial measurement unit (IMU) configured to detect changes in motion and orientation of the cleaning appliance and to generate IMU data in response to the detection. The location determination module 102 can then determine the location data based on the captured video data, captured mapping data and / or captured IMU data.

[0098] In some examples of the systems disclosed herein, there may not be a mobile device. Instead, the location determination module 102, audio detection module 104, display component 106, processor 108 and environment sensing system 110 may be location at the cleaning appliance 110 itself. In further examples, the location determination module 102, display component 106 and / or processor 108 may be located remotely from the cleaning appliance 20. For example, the location determination module 102 and / or processor 108 may be located at a remote server or some other remote device. In these examples, the audio detection module 104 may be configured to wirelessly communicate with the remote location determination module 102 and / or processor 110. The display component 106 may be located at a remote mobile device, and may be configured to wirelessly communicate with the processor 110, for example. This may be useful when a user wants to track the cleaning progress of a robotic vacuum cleaner from a different room, for example. In these examples, the audio detection module 102 and the environment sensing system 110 are still located at or near the cleaning appliance 110 in order to detect the sound of the cleaning appliance operating, and to capture in the mapping data the location of the cleaning portion 22 of the cleaning appliance 20 relative to its surroundings.

[0099] Figure 4 is a flowchart illustrating a method 200 performed by the systems described above. The steps of method 200 may be performed during a cleaning operation of the cleaning appliance 20, for example while the user is using a vacuum cleaner to vacuum the floor of a building. 22 P004482-W001

[0100] To trigger the system 1 to begin the steps of method 200, a user may load the application for performing the method 200 on the mobile device 10. The application may load an augmented reality session whereby the user can track where the cleaning appliance 20 has cleaned, and / or needs to clean, by way of a visual representation of such information on a display screen at the mobile device 10.

[0101] Optionally, as shown in S202 of Figure 4, the environment sensing system 110 (e.g., the camera) may begin capture mapping data of its surroundings. As the environment sensing system 110 is located in close proximity to the cleaning appliance 20 (e.g., attached thereto, forming part of the cleaning appliance 20, or in a mobile device 10 which is coupled to the cleaning appliance 20) the surroundings of the environment sensing system may include at least the cleaning portion 22 of the cleaning appliance 20. As such, the mapping data includes information related to the location of the cleaning portion 22 of the cleaning appliance 20 relative to its surroundings.

[0102] Next, as another optional method step (not shown in Figure 4), the location determination module 110 and / or the processor 108 may determine that the cleaning portion 22 of the cleaning appliance 20 is in contact with the surface to be cleaned (e.g., the floor) based on the mapping data. For example, the location determination module 110 and / or processor 108 may determine from the captured video, LiDAR or IMU data that the cleaning portion 22 is in contact with the floor. In response to the determination that the cleaning portion 22 is in contact with the surface to be cleaned, the audio detection module 104 may be switched on (see S204 of Figure 4). The audio detection module 104 may then begin listening for sounds from the surrounding environment. The processor 108 or the location determination module 102 may instruct the audio detection module 104 to switch on when the cleaning portion 22 is in contact with the surface.

[0103] The audio detection module 104 detects sound from its surroundings, and detects when the cleaning appliance 20 is operating based on the detected sound (S206 of Figure 4). The audio detection module 104 may do this using one or more machine learning models. The audio detection module 104 may be configured to detect when the cleaning appliance 20 is 23 P004482-W001

[0104] operating by inputting data corresponding to the detected sound into a machine learning model trained to detect when a cleaning appliance is operating; and receiving, from the trained machine learning model, an output indicating that the cleaning appliance 20 is operating.

[0105] The training data used to train the machine learning model may include a plurality of captured sound recordings. These sound recordings may be captured by a mobile device similar to mobile device 10.

[0106] The captured sound recordings may include captured sound recordings of one or more different types of cleaning appliances cleaning different types of surfaces. The captured sound recordings may also include captured sound recordings of different background sounds, such as a kitchen blender operating, a dishwasher operating, a human talking, a washing machine operating, a tumble drier operating, a kitchen mixer operating, a radio or other music player playing, a hedge trimmer operating, a lawnmower operating, any other sound other than a cleaning appliance operating. The machine learning model can therefore be trained to distinguish when the cleaning appliance 20 is operating compared to one or more background noises. It can also distinguish between different types of cleaning appliances, different models or makes of cleaning appliance, and / or the different surfaces which are being cleaned.

[0107] In response to the audio detection module 104 detecting that the cleaning appliance 20 is operating, the location determination module 102 determines location information indicative of a location of the cleaning portion 22 of the cleaning appliance 20 over time, as the cleaning appliance moves across the surface (S208). The processor 108 then generates instructions that cause the display component 106 to display a visual representation of the location information (S210), thereby displaying the track of the route of the cleaning portion 22, and thus the areas that have been cleaned. The processor 108 may only generate these instructions in response to the audio detection module 104 detecting that the cleaning appliance 20 is operating. As such, the display component 106 may only render the visual representation of the location information when the cleaning appliance 20 is operating. 24 P004482-W001

[0108] As such, although the location determination module 102 may be capturing mapping data of the surroundings before the audio detection module 104 detects the sound of the cleaning appliance 20 operating, the processor only instructs the rendering of the visual information representing the location of the cleaning portion 22 with respect to the surroundings when the audio detection module 104 detects that the cleaning appliance 20 is operating.

[0109] In some examples, the location determination module 102 may not operate at all prior to the detection that the cleaning appliance 20 is operating. In these examples, the audio detection module 104 may switch on and begin monitoring for sounds when the user first loads the application at the mobile device 10.

[0110] Figure 5 shows an example schematic illustration of a visual representation of the location information displayed at the display component. The systems described above may be adapted to utilize any or all of the various visual representations described herein. As discussed above, the visual representation of the location data may take different forms according to various aspects of the invention.

[0111] Figure 5 shows an illustration of the field of view of a camera capturing the video data, such as the field of view of mobile device 10 mounted to a cleaning appliance 20 as shown in Figure 2, with the visual representation of the location information overlaid thereon.

[0112] In the example shown in Figure 5, the depicted field of view includes the cleaning portion 22 (e.g., a cleaning head 805) of the cleaning appliance 20, a floor surface 810, which is the surface being cleaned, and wall surfaces 815. In addition, Figure 5 shows the visual representation of the location information as a tracked position of the cleaning head 805 of the cleaning apparatus over time in the form of areas 820a, 820b and 825, which represent an area of a single pass of the cleaning head and an area of multiple passes, specifically two passes, of the cleaning head, respectively. Whilst the current field of view is displayed to the user, the full extent of the visual representation may not fit into the field of view visible to the camera, for instance, when the surface being cleaned is the floor of a large 25 P004482-W001

[0113] room. However, the visual representation, for example the tracked location of the cleaning head on the surface, may be persistent outside of the current field of view such that when a given area that has been cleaned re-enters the field of view of the camera, the visual representation previously generated for said area is shown on the display component. In the particular example shown in Figure 5, the visual representation of the location information comprises a coloured track overlaid on the captured video data to represent where the cleaning head of the cleaning appliance has been since it began cleaning the surface. As discussed above, a parameter of the colour may change as the location of the cleaning apparatus moves across the surface being cleaned. The parameter of the colour may comprise one or more of: opacity; hue; saturation; brightness; and the like. For example, where the visual representation of the location data includes a single colour, the colour may become more opaque, more saturated and / or brighter as the cleaning appliance passes over a given position on the surface being cleaned one or more times, meaning that an opaque, saturated and / or bright colour on the display may be indicative of a clean area of the surface. Alternatively, again where the visual representation of the location data includes a single colour, the colour may become less opaque, less saturated and / or darker as the cleaning appliance passes over a given position on the surface being cleaned one or more times, meaning that a translucent, unsaturated and / or dark colour on the display may be indicative of a clean area of the surface. In a further example, where the visual representation of the location data includes multiple colours, the hue of the colour may change as the cleaning appliance passes over a given position on the surface being cleaned one or more times, meaning that a given colour, which may be predetermined or set by the user, may be indicative of a clean area of the surface.

[0114] In the example shown in Figure 5, the user has initiated the cleaning appliance, and the audio detection module has determined that the cleaning appliance is operating. The location determination module is therefore determining the location information and the instructions generated by the processor are causing the display to display the visual representation of the location information. The user has performed two movements of the cleaning head 805 over the floor surface 810. In the first movement, the user has pushed the cleaning appliance to define a first area 820a of the visual representation. In the second movement, the user has pulled the cleaning appliance to define a second area 820b of the 26 P004482-W001

[0115] visual representation. For example, the visual representation of the location information may be the colour blue with an opacity of 20%, meaning that the areas 820a and 820b will be blue tracks with an opacity of 20%, illustrating where the cleaning head 805 has passed over the floor surface 810 being cleaned. When performing the second movement, the user has moved the cleaning head 805 over part of the first area 820a, thereby forming an overlapping region 825 between the two areas. The overlapping region has had multiple passes of the cleaning head 805, whereas the areas 820a and 820b have only had a single pass of the cleaning head. In the example of Figure 5, the overlapping region 825 will simply have the visual representation of the location information compounded, meaning that the overlapping region will be shown as the colour blue with an opacity of 40%. The compounding of the visual representations in overlapping regions such as 825 need not be linear. For example, the overlapping region may be shown as the colour blue with an opacity of 30% for two passes, 35% for three passes and so on.

[0116] While the invention has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art when given this disclosure. Accordingly, the exemplary embodiments of the invention set forth above are considered to be illustrative and not limiting. Various changes to the described embodiments may be made without departing from the spirit and scope of the invention.

[0117] For the avoidance of any doubt, any theoretical explanations provided herein are provided for the purposes of improving the understanding of a reader. The inventors do not wish to be bound by any of these theoretical explanations.

[0118] Any section headings used herein are for organizational purposes only and are not to be construed as limiting the subject matter described.

[0119] Throughout this specification, including the claims which follow, unless the context requires otherwise, the word “comprise” and “include”, and variations such as “comprises”, “comprising”, and “including” will be understood to imply the inclusion of a 27 P004482-W001

[0120] stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

[0121] It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” one particular value, and / or to “about” another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by the use of the antecedent “about,” it will be understood that the particular value forms another embodiment. The term “about” in relation to a numerical value is optional and means for example + / - 10%.

Claims

28 P004482-W001CLAIMS1. A system for tracking areas which have been cleaned using a cleaning appliance, the system comprising:a location determination module; andan audio detection module configured to detect sound from its surroundings, wherein:the audio detection module is configured to detect when a cleaning appliance is operating based on the detected sound; andthe location determination module is configured to, in response to the audio detection module detecting that the cleaning appliance is operating, determine location information indicative of a location of a cleaning portion of the cleaning appliance.

2. The system of claim 1, wherein:the system further comprises a processor, and the processor is configured to generate instructions that cause a display component to display a visual representation of the location information.

3. The system of claim 1 or claim 2, wherein the audio detection module is configured to distinguish the sound of a particular cleaning appliance operating from one or more other different background sounds.

4. The system of any preceding claim, wherein the audio detection module is configured to detect when a cleaning appliance is cleaning a surface based on the detected sound.

5. The system of any preceding claim, wherein the audio detection module is configured to detect when a cleaning appliance is operating by:inputting data corresponding to the detected sound into a machine learning model trained to detect when a cleaning appliance is operating; andreceiving, from the trained machine learning model, an output indicating that the cleaning appliance is operating.29 P004482-W0016. The system of claim 5, wherein the machine learning model is trained using captured sound recordings of one or more cleaning appliances operating and / or captured sound recordings of one or more different background sounds.

7. The system of claim 6, wherein the sampling rate of the audio detection module is approximately equal to the sampling rate of the captured sound recordings used for training the machine learning model.

8. The system of any of claims 5-7, wherein the machine learning model is trained to detect a type of surface being cleaned by the cleaning appliance during operation of the cleaning appliance based on the detected sound.

9. The system of any of claims 5-8, wherein the machine learning model is trained to detect a type of cleaning appliance operating based on the detected sound.

10. The system of any preceding claim, wherein the location determination module is configured to determine the location information at predetermined time intervals.

11. The system of any preceding claim, wherein:the system further comprises an environment sensing system configured to capture mapping data of its surroundings; andthe location determination module is configured to determine the location information based on the captured mapping data.

12. The system of claim 11, wherein:the cleaning component defines a portion of a surface being cleaned at any given time;the location determination module is configured to detect when the cleaning component is in contact with the surface based on the mapping data captured by the environment sensing system; and30 P004482-W001the audio detection module is configured to start detecting sound in response to the detection that the cleaning component is in contact with the surface.

13. The system of claim 11 or claim 12, wherein the environment sensing system comprises one or more ofa camera configured to capture video data of its surroundings; a light detection and ranging (LiDAR) unit configured to capture mapping data of its surroundings; andan inertial measurement unit (IMU) configured to detect changes in motion and orientation of the cleaning appliance and to generate IMU data in response to the detection.

14. The system of any preceding claim, further comprising a mobile device, the mobile device comprising the location determination module and the audio detection module.

15. The system of claim 14, wherein the system further comprises the cleaning appliance, and the mobile device is mountable on the cleaning appliance.

16. The system of any of claims 1-13, further comprising the cleaning appliance, wherein the cleaning appliance comprises the location determination module and the audio detection module.

17. The system of any preceding claim, wherein the cleaning appliance is a floorcleaning appliance.

18. The system of any preceding claim, wherein the floor-cleaning appliance is a vacuum cleaner, a wet vacuum cleaner, a dry and wet vacuum cleaner, a polisher, a steam cleaner, a hard surface cleaner or a carpet cleaner.

19. A method for tracking areas which have been cleaned using a cleaning appliance, the method comprising:31 P004482-W001detecting, by an audio detection module, sound from the surroundings of the audio detection module;detecting, by the audio detection module, when a cleaning appliance is operating based on the detected sound; anddetermining, by a location determination module, and in response to the audio detection module detecting that the cleaning appliance is cleaning, location information indicative of a location of a cleaning portion of the cleaning appliance.

Images