Lung monitoring system

Abstract

Broadly speaking, embodiments of the present techniques relate to a lung monitoring system. In particular, the present techniques relate to a method and device for monitoring a patient's lung, particularly using a portable device. The lung monitoring techniques may then be used to infer information a patient's lung health. In particular, the techniques comprise a computer-implemented method for tracking a user's chest movement. The techniques also comprise a computer- implemented method for stratifying a user into a prognostic group for treatment of a lung condition.

Description

Lung Monitoring SystemTECHNICAL FIELD

[0001] The invention relates to a method and device for monitoring a patient’s lung, particularly using a portable device.BACKGROUND

[0002] Chronic respiratory diseases are the third leading cause of death, accounting for approximately 4 million deaths worldwide in 2019. Respiratory signal collection and analysis play a pivotal role in the early diagnosis and management of respiratory conditions, offering invaluable insights into the functioning of the respiratory system. Spirometry is a known technique which analyses respiration lung flow and volume to determine whether patients have specific lung conditions. Examples of the flow loop patterns for normal, restrictive and various obstructive lung conditions are shown in Figures 1a to 1f. Figure 1a shows the flow loop for a normal patient and this flow loop is also reproduced on each of Figures 1b to 1f as a comparison for the flow loop of the patient with the restrictive condition or the obstructive conditions. Figure 1c shows a pattern from a patient with a general obstructive condition with Figures 1d to 1f showing flow patterns for specific obstructive conditions such as fixed upper airway obstruction, variable extrathoracic obstruction and variable intrathoracic obstruction.

[0003] Figure 2 is a plot of volume of exhaled breath against time which can be accessed using known techniques and compares the results when there is no forced manoeuvre against a forced manoeuvre. However, analysing these sounds often requires one-to-one and / or face-to-face presence to listen to sounds from the lungs. Figure 2 compares the plots for a normal patient (in blue) and a patient with COPD (in red). Various metrics are indicated on Figure 2 including inspiratory reserve volume (AIRV), tidal volume (ATV), expiratory reserve volume (AERV), vital capacity (Ave), inspiratory capacity (Aic), forced expiratory volume in one second (AFEVI) and forced vital capacity (AFVC). These metrics are accessible from experimental data for the patient. However, there are also some useful metrics which are not accessible from experimental data for the patient. These are shown on Figure 2 and include residual volume (RV), total lung capacity (TLC) and functional residual capacity (FRC).

[0004] Signals such as those shown in Figures 1a to 2 encompass a spectrum of physiological data such as airflow, respiratory rate, tidal volume, and oxygen saturation. These signals provide clinicians with crucial information for detecting abnormalities and initiating timely interventions. Early diagnosis is paramount in the realm of respiratory health, as many conditions, if left undetected, can progress to advanced stages, leading to significant morbidity and mortality.

[0005] While patient data is typically assessed by experienced clinicians in classical diagnostic settings, direct access to clinical care is not always possible or even sought by individuals unlessabsolutely necessary. In these contexts, the development of tools, Including for example machine learning (ML)-based tools, that are easily accessible and that can serve as preliminary diagnostic devices to help direct their users to seek medical advice or appropriate therapeutic interventions whenever needed are extremely valuable.

[0006] In this regard, signal-based classification, inherently a time series problem, presents several significant challenges, primarily due to the high dimensionality of the data and the often limited number of recordings available for analysis. The temporal nature of signal data means that each observation consists of numerous sequential measurements, leading to datasets with large numbers of features. This high dimensionality can complicate the modelling process, as traditional algorithms might struggle with the curse of dimensionality, where the volume of the feature space increases exponentially with the number of dimensions, diluting the density of data points. Additionally, signal data is frequently characterised by noise and variability, which further complicates the extraction of meaningful patterns.

[0007] The present applicant has recognised the need for an improved device for sensing and optionally assessing patient information.SUMMARY

[0008] According to the present invention there is provided an apparatus, system and method as set forth in the appended claims. Other features of the invention will be apparent from the dependent claims, and the description which follows.

[0009] In a first approach of the present techniques there is described a computer-implemented method for tracking a user’s chest movement. The method comprises receiving video data of the user’s chest showing the user breathing, the video data comprising a plurality of two-dimensional images showing the user’s chest and locating at least one region of interest in a first image of the plurality of images, the at least one region of interest comprising a plurality of pixels. For each other image in the plurality of images, the method further comprises locating at least one tracking area in the image, wherein the located at least one tracking area corresponds to the at least one region of interest; and for the at least one tracking area, determining a difference in location between the at least one tracking area in the image and the corresponding at least one tracking area in a previous image from the plurality of images, wherein determining the difference in location includes determining the difference in location along a direction perpendicular to a plane defined by the two-dimensional image. The method further comprises determining, using the determined location differences, movement of the user’s chest; and outputting a signal comprising information on the determined movement of the user’s chest.

[0010] Chest movement due to breathing corresponds to three-dimensional movement with the movement in two dimensions (x and y) typically being captured directly in the images. For example, determining a difference in location between the corresponding tracking area in the image and a previous image may further comprise calculating a difference in position between at least one pixel ineach tracking area in a plane of the image. In the present techniques, the method determines movement along a direction perpendicular to a plane defined by the two-dimensional image. In other words, along the z axis. The z-direction movement typically carries the most information. This is because as the lung inflates, any point on the surface of the lung moves both outwards and upwards, similar to an inflating balloon. Accordingly, by targeting this movement, the resulting information on chest movement may be more accurate.

[0011] The plurality of images may be images that are taken consecutively in time. Thus, the difference in location between tracking areas may be determined between consecutive images in the plurality of images. The plurality of images may be taken over a time frame which includes at least one breath of the user. Each region of interest and each tracking area comprises a plurality of pixels in the image and the number of pixels defines the size, for example there may be between 100 to 400 pixels. Each tracking area may have a square shape, such as 10x10 pixels or 20x20 pixels. Alternatively, the tracking areas may have any suitable shape such as an approximately circular shape, or a rectangular shape, or an approximately elliptical shape.

[0012] The method may further comprise receiving depth information, e.g. from a depth capture device, for each of the plurality of images; and determining the difference using the received depth information. Any suitable depth capture device can be used such as the one used on iPhone ™ which is called a TrueDepth camera or sensor.

[0013] Locating the at least one region of interest and / or the at least one tracking area may comprise locating at least one marker which has been positioned on a user’s chest. There may be a plurality of markers, wherein each marker encodes a unique identification pattern. The markers may be any suitable marker which is used by a computer vision system to identify and locate an object. For example the markers may be a 2D barcode and / or a fiducial marker, e.g. an AprilTag which is described for example in “AprilTag: A Robust and flexible visual fiducial system” by Edwin Olson published in IEEE international conference on Robotics and Automation in 2011 or an ArUco marker, as described in " Automatic generation and detection of highly reliable fiducial markers under occlusion" by S. Garrido-Jurado et al. in Pattern Recognition 47, 6 (June 2014), 2280-2292. In this arrangement, locating at least one region of interest may comprise locating one of the plurality of markers which has been positioned on a user’s chest. Locating the corresponding tracking area for each other image in the plurality of images may comprise locating each marker in each of the other images, identifying the unique identification pattern for each marker in each other image, and matching the identified unique identification pattern to the unique identification pattern of the marker of each region of interest.

[0014] When using marker(s); receiving depth information may comprise calculating depth information using the located at least one marker. For example, calculating depth information may comprise calculating a size of the at least one marker in the plurality of images; comparing the calculated size with the known size of the at least one marker; and estimating the depth information based on a difference between the calculated and known sizes. A location of at least one edge and / or at least one corner of the at least one marker may also be determined in the plurality of images.Calculating the depth information may further comprise using determined differences in location of the at least one edge and / or the at least one corner. Calculating the depth information may further comprise using intrinsic parameters of the camera used to capture the image. Such parameters may include for example sensor size and / or focal length.

[0015] Alternatively, calculating depth information may comprise: obtaining a normal vector oriented perpendicular to a surface of the marker for each marker in each of the plurality of images; determining a difference in normal vector orientation and location between images; and estimating the depth information using the determined difference in normal vector orientations and locations. Advantageously, when normal vectors of the markers are used, calculations may be performed in a reference frame of the marker. This means that it is not necessary to calculate distances to the camera. Instead, it is possible to track relative changes in marker position without by comparing markers in consecutive frames without the additional step of calculating properties of the marker relative to camera position. This may also mean that determining the difference in location between images comprises: for the first image, setting the location of the normal vector as a reference location; and for each other image in the plurality of images, determining the difference in location relative to the reference location. That is, an initial location of the normal vector may be set as the origin.

[0016] Locating at least one marker which has been positioned on a user’s chest may comprise locating at least one marker which has been positioned on a user’s chest in a skin conformal way. Skin conformal here means that the marker moves when the user’s chest moves, i.e. that the marker movement is the same or substantially the same as the user’s chest movement. For example, loose fitting clothing may lead to less accurate movement, and thus, attaching the markers to loose fitting clothing also leads to less accurate results. Locating at least one marker which has been positioned on a user’s chest in a skin conformal way may comprise locating at least one marker which has been applied to a user’s hand which is resting on the user’s chest. A user’s hand contacting the user’s chest means that the hand and thus, the marker both move along the user’s chest. This in turn allows for precise tracking as the marker now moves in a skin conformal way.

[0017] The method may further comprise locating a plurality of markers, wherein each marker encodes a unique identification pattern. This may be advantageous for improved accuracy, i.e. tracking information from multiple markers may be combined to achieve overall more accurate tracking. Alternatively, when tracking information provided by a marker is determined to lack the necessary level of accuracy, this information may be discarded and alternative information from the other marker(s) may be used. Again, this improves accuracy. For example, one marker may be located in a lower chest area on the user’s chest and / or on the abdomen, and at least one other marker may be provided in an upper chest area on the user’s chest. This may be useful for monitoring users with different breathing styles. For example, some users tend to breathe into their abdomen, leading to greater movement in a lower chest area. On the other hand, some users tend to breathe into their chest, and thus, more movement can be detected in an upper chest area.

[0018] Defining the at least one region of interest may comprise defining the location of each region of interest within the first image. This may comprise defining a target area in the first image. Using aplurality of tracking areas to track a user’s chest movement allows tracking of individual areas of a user’s chest. The target area may thus be a substantial portion of a user’s chest or a specific area within a user’s chest to more accurately target a part of a user’s lung. Thus, the method may also include finding a target area that includes a substantial portion of the user’s chest. For example at least 10%, preferably between 20% and 90%, more preferably between 20% and 95%, most preferably around 100% of the user’s chest which may then include parts of the abdomen or the shoulders. Alternatively, a substantial portion may mean over 50%, over 75% or over 80% of a user’s chest. The method may further comprise determining whether each tracking area moves synchronously with all other tracking areas and this may allow determination of lung function of or irregularities originating in different parts of the lung.

[0019] There may be a grid of tracking areas and / or regions of interest and / or markers. There may be a grid of 4x4 or 8x8. There may be a number of pixels between each tracking area which are not part of the tracking area and thus separate adjacent tracking areas. Advantageously, this ensures that when a corresponding tracking area is located in a subsequent image, the accuracy of that located tracking area is likely to be improved.

[0020] Locating at least one tracking area in each of the other images may comprise, for each other image in the plurality of images, placing a search area at each location in the other image corresponding to a location of at least one region of interest. The search area may be placed at a location so that an indicative (or initial) tracking area within the search area is at the same location as the corresponding region of interest. The location of the search area which may be determined by coordinates of at least one pixel within the search area or within the region of interest. The search areas may be located at the same locations in each image, or may be updated after a fixed or predetermined number of images have been processed (e.g. 30 or so images).

[0021] The tracking area is likely to have moved from one image to the next and thus each search area may comprise a peripheral area around an indicative tracking area, wherein the peripheral area comprises a fixed number of pixels in each direction around the indicative tracking area. The search area thus comprises the indicative tracking area and peripheral area. The fixed number may be the same (e.g. five) in each direction or different in different directions.

[0022] The video data may comprise all three of the RGB (red, green, blue) colour channels, or it may comprise only one colour channel. Additionally or alternatively, the video data may be infrared video data, i.e. the video may have been recorded at infrared wavelengths. A pixel value may be a value based on one or more colour channels, or based on a value of the infrared videodata. It will be appreciated that the difference score may also be calculated using infrared values. The difference score may be alternatively termed a similarity score.

[0023] The method may comprise determining a confidence level in the located plurality of corresponding tracking areas before determining a difference in location between the corresponding tracking area in the image and a previous image (and doing any further processing). Confidence in the located plurality of corresponding tracking areas may be quantified in an appropriate way. For example, when difference scores are used as described above, the difference scores of the trackingareas between subsequent images may be monitored and compared to a typical difference in pixel values. Only when the confidence level is above a confidence threshold, is further processing for that image continued, e.g. continuing with the determining a difference step. When the confidence is low, the image is rejected and the next, subsequent image in time, is processed.

[0024] The use of search areas which are larger than the tracking areas helps to compensate for movement by the user because such movement means that the physical location on a user’s chest of the tracking areas (and hence in the images) is also likely to have moved. It will be appreciated that updating the search areas after a certain number of images have been processed may provide further compensation to correct for user movement. Merely as an illustrative example, the user may have moved towards the top left direction of the image and thus the subarea that most closely matches the template tracking area is always located in the top left corner of the search area. By recentring the search area on the tracking area, the search areas are more likely to include the tracking area.

[0025] In an alternative embodiment, locating at least one region of interest in a first image of the plurality of images may comprise locating at least one object overlaying the user’s chest; and for each other image in the plurality of images, locating at least one tracking area in the image may comprise locating at least one object overlaying the user’s chest corresponding to the at least one object in the first image. The at least one object may be an object that is easily detectable. For example, the at least one object may be a geometric shape that may have a specific colour that makes the object easily detectable by machine learning or conventional algorithms. Placing such an easily detectable object on the user’s chest may advantageously improve the accuracy of obtaining chest movement data, as the object’s movement may act as a proxy for movement of the user’s chest.

[0026] Advantageously, the at least one object may be one or more of a user’s hand. Using the hands means that it is not necessary to use other objects that may need to be standardised or carried around, such as stickers for placing on a user’s chest. Using the hands is also advantageous because when the hands are placed on the user’s chest, they are substantially skin conformal and thus move along with the user’s chest, meaning that tracking the user’s hands may lead to greater tracking accuracy than, for example, trying to track chest movement through loose fitting clothing. Hands may be detected using any suitable method. For example, machine learning libraries, such as mediapipe (Lugaresi, Camillo, et al. " Mediapipe: A framework for perceiving and processing reality." Third workshop on computer vision for AR / VR at IEEE computer vision and pattern recognition (CVPR). Vol. 2019. 2019.) are capable of detecting body parts with high accuracy and efficiency. These libraries may be leveraged to accurately and efficiently identify body parts, such as hands.

[0027] While there have been described alternative embodiments above, it will be appreciated that each of the methods described as alternatives above may also be used consecutively and / or in parallel and their outputs may be combined in order to achieve high accuracy breathing data.

[0028] Receiving video data of a user breathing may comprise receiving pre-processed video data of a user breathing. The video data may be pre-processed using at least one of bandpass filtering and smoothing. The bandpass filtering may be done using any suitable filter, e.g. a second orderButterworth filter. Similarly, the smoothing may be done using any suitable technique, e.g. a Savitzky-Golay filter.

[0029] The method may further comprise using the determined information on the movement of the user’s chest to extract information about the user’s breathing and / or lung function. The method may further comprise outputting a loop diagram of the user’s breathing, the loop diagram representing a user’s lung function. Lung function may be expressed using any suitable metric, including at least one or more of the following:• Respiratory rate: number of breaths per minute.• Respiratory pattern (e.g. over a fixed time frame such as a minute),• Forced expiratory volume (measured as FEV1, FEV2, FEV3, FEV4, FEV5 and / or FEV6):forced expiratory volume is how much air a person can exhale during a forced breath after 1, 2, 3, etc. seconds,• Forced vital capacity (FVC): forced vital capacity is the total amount of air exhaled during a fast spirometry manoeuvre,• Vital capacity (VC): vital capacity is the total amount of air exhaled during a slow spirometry manoeuvre,• PEFR (peak expiratory flow rate),• Tidal volume,• Minute ventilation: volume ventilated over a minute,• FEF (forced expiratory flow) at 25%, 50% and 75%,• Signal quality: Template matching score (i.e. breath self-similarity),• Properties of a displacement loop as a surrogate of a flow-volume loop,• Displacement loop volume as a surrogate of a flow-volume loop volume, e.g. twice an expiratory volume and / or twice an inspiratory volume, and / or• Displacement loop times, e.g. twice an expiratory time and / or twice an inspiratory time,

[0030] That is, outputting a signal may comprise outputting at least one of: a breathing rate, a loop diagram of the user’s breathing, and / or at least one metric of lung function.

[0031] For example, outputting at least one metric of lung function may comprise: obtaining properties of at least one loop diagram of the user’s breathing, including at least an area under the curve for an inhalation and an area under the curve for an exhalation; calculating a ratio of the area under the curve for an exhalation and the area under the curve for an inhalation; and outputting the calculated ratio. That is, lung function may also be assessed by calculating a ratio of area under the curve for an inhalation and exhalation, i.e.:Area ExhalationArea InhalationThat is, not only the shape of a flow loop diagram, but also an area below the curve changes with an obstructive breathing pattern. Conventionally, area above the x-axis shows exhalation, whereas area below the x-axis shows inhalation. However, it will be appreciated that both of these areas may be referred to as “areas below / under the curve”, as plotting a curve above or below the x-axis is simplyconvention and could easily be switched around. For patients with obstructive pulmonary disease, area under the curve during inhalation is often significantly greater than during exhalation. Thus, the above ratio is smaller for users with an obstructive lung disease than for healthy users. For example, the above ratio may be 0.64 for a user with obstructive lung disease, whereas it may be 0.97 (i.e. close to 1) for a healthy user with no pulmonary issues. Thus, a classification of how likely a user is to have an obstructive pulmonary disorder, such as COPD, may involve calculating the above mentioned ratio of area under the curve for an inhalation and exhalation and comparing this to a reference ratio. For example, a reference ratio may be 0.8, 0.85 or 0.9 and when a ratio calculated for a user is below this reference ratio, the user may be classified as having an obstructive pulmonary disorder. It will also be appreciated that the above ratio may be calculated for several loop diagrams of the user and then averaged. For example, a median, arithmetic average or any other suitable averaging method may be used to calculate an average when more than one ratio is calculated for the same user. When the above ratio is output as a measure of lung health, the averaged ratio may be output.

[0032] In a second approach to the present techniques, there is described a computer-implemented method for stratifying a user into a prognostic group for treatment of a lung condition or respiratory disease. The method may comprise receiving at least one signal which is indicative of movement of a user’s chest; selecting a first binary classifier which outputs a first classification and a second classification; selecting a second binary classifier which outputs a third classification and a fourth classification, wherein at least one of the third and fourth classifications is different from the first and second classifications; extracting at least one feature from the received signal; inputting the at least one extracted feature into at least one of the first and second binary classifiers to obtain a classification of the received signal; and outputting the obtained classification.

[0033] It will be appreciated that the first and second aspects may be used separately or may be combined. For example, the received at least one signal which is used for the stratification of users may have been generated using the method described in relation to the first aspect.

[0034] Respiratory diseases may, for example, include: pneumonia, chronic obstructive pulmonary disease (COPD), other restrictive or obstructive pulmonary diseases and / or interstitial lung disease (ILD). Determining a respiratory disease may also mean determining if the user has an obstructive or restrictive pulmonary disease. Examples of restrictive lung disease are: asbestosis, sarcoidosis, connective tissue diseases, such as scleroderma, drug-induced lung disease, exposure to toxins and pulmonary fibrosis. Examples of obstructive lung diseases are: asthma, bronchiectasis, chronic obstructive pulmonary disease (COPD), and emphysema.

[0035] Processing the received signal to extract at least one feature may comprise processing the received signal to extract at least one shapelet. Before extracting the at least one feature, the received at least one signal may be pre-processed, for example by at least one of filtering and smoothing. The pre-processing may further comprise generating a gradient signal. The at least one feature may be extracted from the processed signal and / or the gradient signal.

[0036] Inputting the at least one extracted feature into at least one of the first and second binary classifiers may comprise inputting the at least one extracted feature into both of the first and second binary classifiers. Depending on the first and second classifications which are output from the first binary classifier, it may only be necessary to input the at least one extracted feature into the first classifier, for example when the first classification is a normal classification (e.g. a healthy patient). The method may further comprise receiving a plurality of signal which are indicative of movement of a user’s chest; obtaining a classification for each of the plurality of signals; and deriving, from the obtained classifications, a single classification for the user. It will be appreciated that some or all of the obtained classifications may be different even though the signals were extracted from the same user. Deriving a single classification may be done using any suitable technique, including for example selecting a most common obtained classification as the single classification or selecting a classification with a highest value as the single classification. Highest value may be a highest probability or likelihood score for the classification.

[0037] Selecting the first and second binary classifiers may be done so that the user is stratified into one of the following prognostic groups: normal, restrictive and obstructive. For example, the first binary classifier may be a classifier which outputs a first classification of normal and a second classification of abnormal; and the second binary classifier may be a classifier which outputs a third classification of obstructive and a fourth classification of restrictive. Alternatively, the first binary classifier may be a classifier which outputs a first classification of restrictive and a second classification of not restrictive; and the second binary classifier may be a classifier which outputs a third classification of obstructive and a fourth classification of normal. It will be appreciated that other combinations of first and second classifiers may be used.

[0038] The stratification of the user into a prognostic group may be used in a method of diagnosis for the user (who may be termed a patient or subject and the terms may be used interchangeably). Alternatively, the information about the user’s chest movement may be analysed differently in a method of diagnosis. Thus, in a third approach of the present techniques, there is a method of diagnosis comprising at least the method of stratifying a user into a prognostic group and / or the method of tracking a user’s chest movement as described above.

[0039] The stratification of the user into a prognostic group may be used in a method of treatment for the user. Alternatively, the information about the user’s chest movement may be analysed differently as part of a method of treatment. Thus, in a fourth approach to the present techniques, there is provided a method of treatment of a subject comprising at least the method of stratifying a user into a prognostic group and / or the method of tracking a user’s chest movement as described above. In other words, the method may comprise obtaining information about a user’s chest movement according to the methods of the first approach; determining, using the extracted chest movement information that a user has a respiratory disease; selecting a treatment according to the determined respiratory disease; and administering the treatment. Similarly, the method may comprise stratifying a user into a prognostic group for treatment of a respiratory disease, selecting a treatment according to the determined prognostic group; and administering the treatment.

[0040] As described above, respiratory diseases may, for example, include: pneumonia, chronic obstructive pulmonary disease (COPD), other restrictive or obstructive pulmonary diseases and / or interstitial lung disease (ILD). Determining a respiratory disease may also mean determining if the user has an obstructive or restrictive pulmonary disease. The treatment for obstructive lung disease may, for example, be selected from any of: bronchodilators, including short-acting bronchodilator inhalers, such as beta-2 agonist inhalers and antimuscarinic inhalers, and long-acting bronchodilator inhalers, such as beta-2 agonist inhalers and antimuscarinic inhalers, steroids, including steroid inhalers and steroid tablets, theophylline, aminophylline, mucolytic medicines, such as carbocisteine and acetylcysteine, antibiotics, nebulised medicines, roflumilast, oxygen therapy, including long-term oxygen therapy and ambulatory oxygen therapy, non-invasive ventilation, and surgery, including a bullectomy, lung volume reduction surgery and a lung transplant. Treatment may also include exercises and education such as pulmonary rehabilitation, improving muscle strength. The treatment for restrictive lung disease may, for example, be selected from any of: inhalers, expectorants, antibiotics, oxygen therapy, chemotherapy, mycophenolate mofetil, rituximab, corticosteroids, other immunosuppressing and anti-inflammatory medications, antifibrotic medications, such as pirfenidone or nintedanib, and lung transplantation. Treatment may also include breath conditioning techniques, such as pursed-lip breathing, slow, deep breathing, or diaphragmatic breathing, upper and lower limb strengthening and conditioning exercises, respiratory muscle strengthening exercises, level walking, relaxation or visualized meditation, eating a balanced nutritious diet, quitting smoking, avoiding environments with toxins, irritants, and allergens that may worsen symptoms.

[0041] According to another aspect of the present techniques, there is described an apparatus for tracking a user’s chest movement, the apparatus comprising: an image capture device for capturing video data of the user’s chest that shows the user breathing; and at least one processor, coupled to memory, and arranged for: carrying out the method described above. The at least one processor may further be configured to stratify a user into a prognostic group for treatment of a lung condition using the method described above. The apparatus may further comprise a user interface for directing the user to position the image capture device for capturing the video of the user’s chest, e.g. in the target area as defined above.

[0042] The apparatus may be a user device, such as a smartphone, tablet computer, smartwatch, computer, laptop or any other suitable user device which comprises both the image capture device and a processor for locally processing the captured video data. Alternatively, the apparatus may comprise a user device which comprises the image capture device and a communication module to send the captured video data to a server or any other suitable device comprising a processor. For example, the apparatus may be integrated into other devices and / or vehicles, such as cars, lorries, vans, busses, or any other vehicle that can travel across roads, water or in the air.

[0043] In a further approach to the present techniques, there is described a non-transitory carrier carrying a computer program which when implemented on a computer, e.g. the user device and / or remote processing device, causes the computer to implement the method described above.BRIEF DESCRIPTION OF DRAWINGS

[0044] For a better understanding of the invention, and to show how embodiments of the same may be carried into effect, reference will now be made, by way of example only, to the accompanying diagrammatic drawings in which:

[0045] Figures 1a to 1f plot flow (L / sec) against volume (L) for a normal patient, a patient with a restrictive condition and patients with different obstructive conditions respectively;

[0046] Figure 2 plots volume of exhaled breath against time for patient undergoing no forced manoeuvre and patients undergoing a forced manoeuvre;

[0047] Figure 3a is a block diagram of an apparatus fortracking a user’s chest movement;

[0048] Figure 3b is a close up of a component in the apparatus of Figure 3a;

[0049] Figure 4 is a flowchart showing a pipeline for analysing a video showing a user breathing;

[0050] Figure 5a is a diagram showing how the user’s chest movement is tracked using a grid of tracking areas;

[0051] Figures 5b to 5d are alternative methods fortracking movement of a user’s chest;

[0052] Figures 5e to 5g show different methods of tracking a user’s chest movement;

[0053] Figure 6a shows an image of a user’s chest with a tracking area;

[0054] Figure 6b shows the image of Figure 6a as captured by a depth capture device;

[0055] Figures 6c and 6d plot movement of the tracking area in Figure 6a in the y and z axes respectively;

[0056] Figure 6e shows an example plot of the white pixel count per frame when the tracking area is the whole of the user’s torso as shown in Figure 5c;

[0057] Figures 6f and 6g show the variation in the z-location for two specific markers;

[0058] Figures 6h and 6i show an example plot of tracking movement of a user’s hand placed on the user’s chest and a detrended plot of tracking movement of a user’s hand placed on the user’s chest respectively;

[0059] Figure 7 is a flowchart showing how the grid of tracking areas is used to determine the user’s chest movement;

[0060] Figure 8a shows a guide outline in an application interface which is used to direct the user to position an image capture device of the apparatus to capture the video data of the user’s chest;

[0061] Figures 8b to 8f illustrate how the location of a tracking area is determined from one image of the video to the next to determine the user’s chest movement;

[0062] Figure 9 is a flowchart showing how the detected movement can be transformed into output breathing information;

[0063] Figure 10a shows a breathing signal of the user plotting movement against time;

[0064] Figure 10b shows a smoothed breathing signal derived from Figure 10a;

[0065] Figure 10c shows a gradient signal of the smoothed breathing signal of Figure 10b;

[0066] Figures 11a to 11 f are loop diagrams corresponding to individual breaths of a user which may be extracted from a gradient signal such as that shown in Figure 10c;

[0067] Figures 12a and 12b combined form a flowchart showing the steps of training a machine learning model;

[0068] Figure 13a to 13c show the distribution of individuals by respiratory condition, gender, and manoeuvre recordings for the data used in the training method of Figures 12a and 12b;

[0069] Figure 13d is a histogram showing the number of signals collected per individual for the two manoeuvres (TB and FS) for the data used in the training method of Figures 12a and 12b;

[0070] Figure 14a plots value against time for a time series with an overlaid shapelet;

[0071] Figure 14b plots distance against time between a shapelet and a time series; and

[0072] Figure 15 is a flowchart for using the trained model of Figures 12a and 12b.DETAILED DESCRIPTION OF DRAWINGS

[0073] Figure 3a is a block diagram of the components of a system which may be used to track a user’s chest movement and hence determine lung function. The system comprises a user device 100, e.g. a smart phone, mobile phone, tablet or similar computing or electronic device which is typically handheld. The user device 100 comprises the standard components such as a processor 102, a user interface 104, storage 106 and a communications interface 108. The user interface 104 may be any standard interface, including a touch sensitive display screen, voice input, keyboard input etc. Similarly, the storage 106 may be conventional memory which may include RAM and / ROM. The user device 100 comprises an image capture device 110 which may be any standard image capture device, such as a camera. The image capture device 110 may be a pixel camera system which uses image sensors and processing techniques to capture high-quality videos. Such a system relies on the pixels in the sensor array combined with powerful computational photography and video processing software.

[0074] The video data may be analysed by the processor 102 on the user device 100, e.g. using a machine learning model 114 as described in more detail below. Alternatively, the raw video data may be transferred, e.g. using the communications interface 108 to a processing module 120. The processing may optionally use a machine learning model 128 on the processing module 120. For example, the communications interface 108 may connect via a wireless connection, e.g. via the Internet. The processing module 120 may be remote from the user device, e.g. located in a different location from the user device or may be at least partially local to the user device (e.g. located within the user device). Using a remote processing module 120 may allow access to more powerful resources so that the raw data may be processed more quickly. A local processor has the advantage that it can be used in areas without any wireless connections. Accordingly, both local and remote processing may occur. The processing module 120 may be implemented in hardware e.g. as a computing device such as a server or may be implemented in the cloud 132. As schematically illustrated in Figure 3a, the processing module 120 comprises a processor 122, at least one filter 124and a communications interface 126 via which the processing module 120 can receive data from the user device 100 and / or send messages to the user device 100.

[0075] As shown in Figure 3a, the system may also comprise a repository 130 into which raw and / or processed data can be stored. The repository 130 may be a secure database for the video files. Data may be transferred from the processing module 120 to the repository 130 or from the cloud 132. Further processing may be done in the cloud 132 and thus the processing module 120 is able to communicate with the cloud 132. The repository 130 may include a secure personalised data store in which the uploaded data may only be accessed by use of a key or other personalised identifier.

[0076] In some instances, a trained machine learning, ML, model (not shown) may optionally be used to analyse the video data. The video data may be analysed to extract information about a user’s breathing. The information about a user’s breathing may further be used to determine if the user’s breathing is within a normal range, and information about the user’s breathing may be output to the user using the user interface 104. The methods used to analyse the video data are described in more detail below.

[0077] The user device 100 may also be any other device suitable for capturing video data of a user’s chest and using the captured video data to track a user’s chest movement. For example, the user device 100 could be a vehicle, such as a car, bus, van, lorry, aircraft, boat or any other vehicle that is equipped with an image capture device. In this example, the image capture device of the vehicle may capture video data of a driver or other occupant and analyse this video data. The vehicle may comprise a processor coupled to memory, capable of doing this analysis. The processor may analyse the video data to track the driver’s or other occupant’s chest movement. This analysis may then be used to infer information about the driver’s or other occupant’s breathing, which may in turn be used to deduce physiological information about the driver or other occupant, such as whether they are tired or not. The user device 100, when part of a vehicle, may then additionally be configured to alert the driver or other occupant if an unfavourable physiological state is detected. For example, if the system detects that the driver is tired and should take a break, it can alert the driver and / or other occupants of the vehicle of this fact.

[0078] Optionally, the user device 100 may comprise a depth capture device 112 which can determine depth within an image. Figure 3b shows an example of the components which may be used to form the depth capture device 112. For example, the depth capture device 112 may comprise an infrared camera 140, a flood illuminator 142 and a dot projector 144 to create a map which includes depth within the captured image. Typically, the depth capture device 112 is used to create a detailed 3D map of a user’s face, e.g. for security purposes. The dot projector 144 is used to project a large number of invisible infrared dots (e.g. 30,000) on the object in the image (e.g. the user’s face). The infrared camera 140 is used to capture the reflected pattern. The flood illuminator 142 may be used to provide additional lighting to ensure that the data can be processed even in low light conditions. Because the size of the projected dots at specific distances is known, the reflected pattern can be used to create a depth map of the captured image. Additionally, any distortion of projected dots in the reflected pattern can be used to determine, for example, an inclination of the surface onwhich the dots are being projected. Any suitable depth capture device 112 can be used such as the one used on iPhone ™ which is called a TrueDepth camera or sensor. The reflected pattern is processed by a machine learning model, e.g. a proprietary neural engine from Apple ™ and can be used for secure facial recognition (Face ID), depth-enhanced portrait photography and real-time facial tracking for features when generating avatars or using augmented reality applications.

[0079] Figure 4 is a flowchart showing a pipeline for analysing a video showing a user breathing. The method shown in Figure 4 may be implemented by the apparatus of Figure 3a. In a first step, video data of the user’s chest showing the user breathing is received S102. The video data may comprise a plurality of images showing the user’s chest. The plurality of images may be images that are taken consecutively in time, such that they show at least one breath of the user over time. The video data may comprise all three of the RGB (red, green, blue) colour channels, as is conventional. Alternatively, only one of the colour channels may be used in the following analysis. For example, the G (green) colour channel may carry more relevant information than the other two colour channels and thus the following analysis may be performed on the G channel of the video data only.

[0080] In a next step, a plurality of tracking areas is defined in the first image of the plurality of images S104 and their location in the first image may be determined. Each tracking area may comprise a plurality of pixels in the image. The location may be defined by the pixel row and column location of at least one of the outermost pixels within the tracking area. For example, the location may be defined by the row and column location of the top-left pixel of each tracking area. The tracking areas may have any suitable size, for example, there may be between 100 to 400 pixels in each tracking area. The shape of each tracking area may be defined by the arrangement of the pixels. For example, the pixels may be arranged in a grid (e.g. 20x20 pixels or 10x10 pixels) and thus each tracking area may have a square shape. Alternatively, the tracking areas may have any suitable shape such as an approximately circular shape, or a rectangular shape, or an approximately elliptical shape.

[0081] The plurality of tracking areas may be arranged in any suitable pattern or may be arranged randomly. The tracking areas may be arranged to avoid overlap between the tracking areas, e.g. by defining a minimum number of pixels between each tracking area. Figure 5a is a schematic diagram of a user 200 showing a grid of tracking areas 206 which are arranged across a target area, in this case a user’s chest 202. In this example, there are sixteen tracking areas 206 arranged in a 4x4 grid 204 but it will be appreciated that this is just one example and a grid of 8x8 tracking areas, or any other suitable grid arrangement may be used. In this example, the tracking areas 206 are spaced equally across the user’s chest area, although this is not necessary and may change if a particular area of the user’s chest is being monitored.

[0082] It will be appreciated that the size of the grid may be determined by a number of factors, such as desired accuracy, or resource constraints of the apparatus on which the method may be implemented. For example, a grid comprising a larger number of tracking areas may correspond to greater accuracy but may also require more computational resources. Thus, the size and nature ofthe grid may be adapted to achieve a desired accuracy, or to function within the limitations of a resource constrained device.

[0083] It will also be appreciated that there is a relationship between the size and nature of the grid and the size and spacing of the tracking areas within the grid. For example, for efficient detection of tracking areas in subsequent images, it may be necessary for the tracking areas to comprise a certain number of pixel values. If the grid contains a larger number of tracking areas, this may mean that each tracking area contains fewer pixels to avoid any overlap between the tracking areas. This might make detecting tracking areas in a subsequent image more difficult and less accurate. Thus, it will be appreciated that there may be a balance to be struck between higher resolution (i.e. a larger number of tracking areas) and maintaining accuracy.

[0084] Figures 5b and 5c show alternative arrangements in which there is only a single tracking area 220 rather than a grid of tracking areas. Figures 5b and 5c show an image frame in a selected plane and show the x-axis pixels against the y-axis pixels. In the Figure 5b example, the single tracking area is relatively small compared to the overall image, for example a small number of pixels. The tracking area 222 can be selected to be an arbitrary size and can encompass the whole of a user’s chest (torso) as shown for example in Figure 5c. There can also be any number of tracking areas.

[0085] Figure 5d shows another alternative arrangement in which a grid of markers 230 is used to assist in locating the tracking area. These markers 230 are positioned on a user’s chest, for example by a user wearing a bespoke item of clothing in which the markers are printed or otherwise attached to the clothing. Alternatively, the markers 230 may be individually attached by a user. The markers 230 may be any suitable marker which is used by a computer vision system to identify and locate an object. For example the markers 230 may be a 2D barcode and / or a fiducial marker, e.g. an AprilTag which is described for example in “AprilTag: A Robust and flexible visual fiducial system” by Edwin Olson published in IEEE international conference on Robotics and Automation in 2011. Another suitable fiducial marker are ArUco markers, as described in " Automatic generation and detection of highly reliable fiducial markers under occlusion" by S. Garrido-Jurado et al. in Pattern Recognition 47, 6 (June 2014), 2280-2292. AprilTags, ArUco markers or similar markers are typically useful for applications like robot navigation, object tracking, and augmented reality.

[0086] The marker 230 may be a regular shape, e.g. a square or rectangle, with edges and corners which are readily detected. Each marker 230 has a unique pattern (e.g. pattern of black and white squares) which encodes a specific and unique identification (ID) for that marker 230. This identification can be determined by the computer vision system. The physical dimensions, e.g. size, of the marker 230 are predetermined and thus known, for example from the determined ID. When a marker is detected in the image, the location of the marker within the image may be defined using the location of the outermost pixels. Moreover, the apparent size in pixels of the detected marker 230 can be determined and compared with the predetermined size. This may enable the perspective geometry for the marker 230 to be determined. The further the marker 230 is from the camera, the smaller it appears in the image (and vice versa). In other words, the focal length of the camera and / or other intrinsic parameters of the camera, may be used to relate the determined pixel size of the marker toits real-world distance. Thus, the z motion across the torso can be tracked using the markers. Furthermore, using the edges and corners of the tag, the system can also estimate orientation (tilt) and distance for each marker 230.

[0087] The marker-based respiratory monitoring system utilizes a novel approach to tracking chest and abdominal wall motion through specialized markers designed for optimal detection and tracking in three-dimensional space. The markers may be implemented in various geometric patterns including squares, triangles, hexagons, or pentagons, with each shape offering distinct advantages for granular motion tracking. The size of individual markers may range from approximately 0.1-1 centimetre to 5 to 10 centimetres in width, with the specific size selected based on factors including the expected distance from the camera, required tracking precision, and the degree of flexibility needed for the particular application.

[0088] Each marker incorporates a unique identification encoding pattern that enables individual tracking of specific markers while simultaneously providing orientation detection and scale reference for distance calculations. The patterns are specifically optimized for high contrast visibility under varying lighting conditions and are designed to maintain pattern integrity even when the underlying fabric stretches or flexes during respiratory movements. This robust pattern design ensures reliable detection across a wide range of operating conditions.

[0089] The markers may be implemented through multiple manufacturing methods to suit different use cases. In one implementation, the markers are directly printed onto fabric using specialized inks designed to maintain pattern integrity during stretching and movement. Alternatively, as shown in Figure 5e, the markers may be implemented as adhesive stickers, which may be either rigid or flexible depending on the application requirements. In another implementation, the markers may be integrated into the garment as buttons or specialized fabric elements. Additional implementation methods include heat-transferred patterns or markers that are woven or embroidered directly into the fabric structure.

[0090] The markers may be incorporated into various carrier mediums designed for different measurement scenarios. These carriers include elastic bands or garments specifically designed for placement on the upper chest, lower chest / abdominal region, or full torso coverage. When implemented as bands, the width may range from 1 to 5 centimetres to provide focused measurement zones while maintaining comfort and practicality. The carrier materials incorporate four-way stretch properties to ensure proper conformity to body contours while including moisture-wicking properties for user comfort. The materials are selected to maintain durability and preserve marker visibility even after repeated washing cycles.

[0091] The markers may be implemented across various dimensional ranges to optimize visibility, tracking accuracy, and user comfort. In a primary embodiment, the base marker dimensions may be selected from the following ranges:

[0092] Small Markers:- Width / Height: 8mm to 15mm- Minimum pattern feature size: 0.5mm to 1mm- Border thickness: 0.2mm to 0.5mm- Inter-marker spacing: 15mm to 25mm- Optimal viewing distance: 0.1 m to 1 m, preferably 0.01 m-0.1 m

[0093] Medium Markers:- Width / Height: 15mm to 30mm- Minimum pattern feature size: 1 mm to 2mm- Border thickness: 0.5mm to 1mm- Inter-marker spacing: 25mm to 50mm- Optimal viewing distance: 1m to 2m

[0094] Large Markers:- Width / Height: 30mm to 50mm- Minimum pattern feature size: 2mm to 4mm- Border thickness: 1 mm to 2mm- Inter-marker spacing: 50mm to 100mm- Optimal viewing distance: 2m to 4m

[0095] Band Specifications:- Band width (narrow): 25mm to 50mm- Band width (standard): 50mm to 100mm- Band width (wide): 100mm to 150mm- Elastic stretch factor: 120% to 180% of resting length- Marker zone coverage: 60% to 80% of band surface area

[0096] Pattern Density Specifications:- Minimum marker separation: 1.5x marker width- Maximum marker separation: 3x marker width- Pattern repetition interval: 4 to 8 unique markers before repetition - Vertical marker rows: 1 to 3 depending on band width- Horizontal marker spacing: 2x to 4x marker width

[0097] Anatomical Placement Measurements:- Upper chest placement: 5cm to 10cm below clavicle- Lower chest placement: 10cm to 15cm below axilla- Abdominal placement: 5cm to 10cm above umbilicus- Band overlap allowance: 25mm to 50mm- Band tension indicator marks: every 50mm

[0098] Material Thickness Specifications:- Base fabric thickness: 0.3mm to 0.8mm- Printed marker thickness: 0.1 mm to 0.3mm- Adhesive marker thickness: 0.2mm to 0.5mm- Woven marker thickness: 0.3mm to 0.6mm- Total maximum thickness: <2mm for comfort

[0099] Marker Distortion Tolerances:- Maximum stretch deformation: 30% of original dimensions- Maximum rotational tolerance: ±15 degrees- Maximum perspective distortion: 45 degrees from normal- Pattern recognition threshold: 70% pattern match- Minimum contrast ratio: 3:1

[0100] These dimensional specifications provide a comprehensive framework for implementing the marker system across various applications while maintaining reliable tracking performance. The dimensions may be adjusted within these ranges based on specific use cases, viewing distances, and required measurement precision.

[0101] The marker system enables comprehensive tracking of respiratory motion across all three spatial dimensions. This includes lateral expansion (x-axis movement), vertical displacement (y-axis movement), and anterior-posterior motion (z-axis movement). The markers may be strategically placed to capture various aspects of respiratory mechanics, including upper chest breathing patterns, diaphragmatic breathing movements, asymmetric chest wall motion, and paradoxical breathing patterns. This comprehensive tracking enables detailed analysis of respiratory patterns and potential anomalies. A key technical innovation of the system lies in its ability to maintain accurate measurements during body movement and tissue deformation. The flexible marker system enables continuous respiratory monitoring without requiring rigid attachments, while still maintaining high-precision 3D motion tracking using standard camera systems. The pattern recognition algorithms are specifically designed to remain robust during tissue deformation and movement, ensuring consistent measurements across a wide range of breathing patterns and body positions.

[0102] Figures 5e to 5g show different methods of tracking a user’s chest movement. Figures 5e to 5g show a method in which at least one object overlaying a user’s chest is tracked as the at least one tracking area. The at least one object may be an object that is easily detectable. For example, the at least one object may be a geometric shape that may have a specific colour that makes the object easily detectable by machine learning or conventional algorithms. Placing such an easily detectable object on the user’s chest may advantageously improve the accuracy of obtaining chest movement data, as the object’s movement may act as a proxy for movement of the user’s chest.

[0103] In Figure 5e, the at least one object is the user’s hand. Using the hands means that it is not necessary to use other objects that may need to be standardised or carried around, such as stickers for placing on a user’s chest. Using the hands is also advantageous because when the hands are placed on the user’s chest, they are substantially skin conformal and thus move along with the user’s chest, meaning that tracking the user’s hands may lead to greater tracking accuracy than, for example, trying to track chest movement through loose fitting clothing. Hands may be detected using any suitable method, such as the mediapipe library described above which can accurately and efficiently identify body parts, such as hands.

[0104] Figure 5f shows a method using the markers, as described with reference to Figure 5d above. Using markers may increase the accuracy of the methods used. Advantageously, the markersmay be placed on the back of the user’s hands which are then placed on the user’s chest. Figure 5f shows a method in which markers are placed on the back of user’s hand overlaying the chest. Advantageously, locating at least one marker which has been positioned on a user’s chest may comprise locating at least one marker which has been positioned on a user’s chest in a skin conformal way. Skin conformal here means that the marker moves when the user’s chest moves, i.e. that the marker movement is the same or substantially the same as the user’s chest movement. For example, loose fitting clothing may lead to less accurate movement, and thus, attaching the markers to loose fitting clothing also leads to less accurate results. Locating at least one marker which has been positioned on a user’s chest in a skin conformal way may comprise locating at least one marker which has been applied to a user’s hand which is resting on the user’s chest. A user’s hand contacting the user’s chest means that the hand and thus, the markers both move along the user’s chest. This in turn allows for precise tracking as the marker now moves in a skin conformal way.

[0105] The system may be implemented in various forms to suit different use scenarios. For clinical settings, single-use adhesive marker sets may be employed to ensure hygiene and ease of application. For home use, washable garments with permanent markers provide a durable and reusable solution. Sport-specific designs may integrate the markers into athletic wear, while medicalgrade versions may incorporate additional features for clinical monitoring. Pediatric versions may utilize smaller markers and child-friendly patterns while maintaining the same technical capabilities.

[0106] The marker-based respiratory monitoring system represents a significant advancement in non-invasive respiratory monitoring by combining innovative marker designs with flexible implementation methods and robust tracking capabilities. The system's ability to provide accurate three-dimensional tracking of respiratory motion while accommodating natural body movement and deformation represents a novel approach to respiratory monitoring that is both practical and precise.

[0107] In additional embodiments of the present techniques, the marker system may be implemented through various alternative approaches that enhance its utility and robustness. The markers may be implemented using photochromic materials that change visibility under specific lighting conditions, allowing for dynamic marker activation when measurements are needed while remaining discreet at other times. Such photochromic markers may be activated by specific wavelengths of light from the monitoring device, enabling selective activation of different marker sets for various measurement protocols.

[0108] The marker band may incorporate multiple layers with different elastic properties to optimize both comfort and measurement accuracy. For example, a base layer may provide primary elastic support while an outer layer carries the markers, with an intermediate layer providing cushioning and moisture management. The multi-layer approach may also enable the incorporation of additional sensing modalities, such as integrated stretch sensors or conductive threads that provide complementary measurements to the optical marker tracking.

[0109] In some implementations, the markers may be designed with retroreflective properties that enhance visibility under low light conditions. The retroreflective elements may be integrated into the marker patterns themselves or may form a border around traditional markers to aid in initialmarker location and tracking. These retroreflective properties may be achieved through microspherebased materials, corner-cube retroreflectors, or other retroreflective technologies that maintain their properties when implemented on flexible substrates.

[0110] The marker band may also incorporate registration features that ensure proper placement and orientation on the user's body. These registration features may include anatomical landmarks, asymmetric marker patterns that indicate correct orientation, or physical features such as tactile indicators that help users position the band correctly. The registration features may be designed to be easily detectable by both the user and the monitoring system, ensuring consistent placement across multiple measurement sessions.

[0111] Alternative attachment mechanisms may be employed beyond traditional elastic bands. For example, the markers may be incorporated into adhesive patches that conform to body contours while maintaining precise marker positioning. These patches may utilize medical-grade adhesives that allow for extended wear while being gentle on the skin. In another implementation, the markers may be incorporated into garments with targeted compression zones that ensure stable marker positioning while maintaining user comfort.

[0112] The marker patterns may be designed to incorporate redundancy and error checking capabilities. For example, each marker may include error-checking features that allow the system to verify the marker's integrity and detect any distortion or occlusion. The markers may also be arranged in patterns that maintain tracking capability even if some markers become temporarily obscured or distorted, utilizing mathematical relationships between marker positions to maintain measurement accuracy.

[0113] In some embodiments, the markers may be implemented using materials with specific spectral properties that enable multi-spectral imaging. By utilizing markers that are visible in different spectral bands (e.g., visible light, near-infrared, etc.), the system can employ multi-spectral imaging techniques to improve marker detection and tracking reliability. This approach may also enable the system to differentiate between markers and potential confounding patterns in the background or on clothing.

[0114] The marker system may also incorporate dynamic markers that can change their patterns or properties based on measurement requirements. This may be achieved through various technologies such as e-ink displays, thermochromic materials, or mechanically adjustable patterns. Such dynamic markers could enable adaptive measurement protocols that optimize marker visibility and tracking based on environmental conditions or specific measurement requirements.

[0115] For applications requiring extended monitoring, the marker band may incorporate features that enable marker regeneration or self-cleaning. This may include protective coatings that resist marker degradation, self-healing materials that maintain marker integrity over time, or designs that facilitate easy marker renewal through user-applicable treatments or replaceable marker elements.

[0116] The system may also utilize hybrid approaches that combine traditional markers with projected patterns. In such implementations, a base set of physical markers may be augmented bypatterns projected onto the user's body or clothing, enabling dynamic adjustment of the measurement area and resolution while maintaining the benefits of physical markers for primary reference points.

[0117] These alternative implementations and extensions significantly enhance the robustness and versatility of the marker-based monitoring system, providing multiple layers of protection for the core invention while enabling adaptation to various use cases and environmental conditions. The combination of physical design elements, material properties, and measurement strategies creates a comprehensive system that maintains accuracy and reliability across a wide range of applications and operating conditions.

[0118] Returning to Figure 4, the next step S106 is to determine a difference in location for each tracking area in subsequent images. The difference in location of tracking areas between two consecutive images of the plurality of images corresponds to movement of the user’s chest during the time frame in which the images were captured. By tracking the locations of the plurality of tracking areas across the plurality of images comprising the video data, the user’s chest movement may be determined.

[0119] The next step S108 is to determine the movement of a user’s chest using the determined location differences. Chest movement due to breathing corresponds to three-dimensional movement with the movement in two dimensions (x and y) typically being captured directly in the images and the movement of the chest in the perpendicular z direction (third dimension) being approximated using the two-dimensional movement captured in the image or captured using the depth capture device (e.g. the TrueDepth camera) or determined using the tracking of the markers as described above.

[0120] As an example, Figure 6a shows an image of a user’s chest with an example tracking area 320 and illustrates that the tracking area is tracked in one of both of the x and y directions. The image shown in Figure 6a is captured using a standard image capture device (e.g. a camera). Figure 6a shows that the location of the tracking area can be expressed as the location of the pixels in the x and y axis respectively. Figure 6b captures the same object as Figure 6b but shows a depth map captured using a depth camera device (e.g. a TrueDepth sensor). The same tracking area 320 is marked on Figure 6b and in this example the location of the tracking area in the z-axis is monitored. Figures 6c and 6d are plots of location against time for the y-axis measurement (Figure 6c) and the z-axis measurement (Figure 6d). As shown in Figures 6c and 6d there is an approximately cyclical variation in the y and z vales and a general increase overtime.

[0121] In the example of Figures 5b and 5c, the user is wearing a specific colour of clothing, e.g. white in this example. Accordingly, the number of specific-coloured pixels (e.g. white pixels) is counted as part of the determining of the movement of the user’s chest. The counting may be done by setting a threshold on the grayscale value so that pixels which have a value which is greater than or equal to the threshold are considered to be white and all other pixels are discarded. In other words, there is a binary mask applied to each frame. Surface area of the chest, which is determined by counting a number of pixels of a specific colour in this example, is directly proportional to chest / lung volume. Therefore, simply counting a number of pixels of a predetermined colour, e.g. white, either in the whole of each image frame or in a specific tracking area can be used to determine a lung volumechange. When only looking at a particular tracking area on a user’s chest, it is also possible to determine irregularities in breathing between different areas of the chest, and therefore, lung. For example, at least one tracking area may be placed on the left and right sides of a user’s chest. When counting pixels in these tracking areas, it is possible to monitor if changes in lung volume are equal on both the left and right side, or if there is a difference which may point to a lung disease. When only part of the user’s chest is tracked, a tracking area needs to be placed such that the tracking area only covers part of the user’s chest area. This is necessary to detect a change in pixels of a certain colour which is different from the background colour. In this case, the number of pixels of a certain colour which is being counted changes because the edges of the user’s chest move when the user is breathing. When the user’s chest expands, the edges of the user’s chest move closer to the edges of the captured image and consequently the number of pixels which are being counted increases. When pixels of a certain colour are counted across the whole of the captured image, background will automatically be included and therefore, the image will also include edges of the user’s chest which allows for tracking of changes in the number of pixels of the same colour. Figure 6e shows an example plot of the white pixel count per frame when the tracking area is the whole of the user’s torso as shown in Figure 5c, in other words, tracking the number of pixels over time.

[0122] In the example of Figure 5d, the user is wearing various markers which denote the tracking areas. Figures 6f and 6g show the variation in the z-location for two specific markers, it will be appreciated that similar plots can be found for all separate markers. The variation in the z-axis is determined by combining various factors, such as the known physical size of the marker, the apparent size (in pixels) of the marker in a specific image, the camera’s intrinsic parameters (such as focal length and sensor size) and perspective geometry. By combining these factors, an application which may run on the user device (or elsewhere) calculates the marker's distance from the camera. This process is called pose estimation and often uses mathematical models to extract 3D position and orientation data based on the marker's 2D image. The plots for each marker may be combined to generate an overall measurement or different areas of the chest can be tracked separately.

[0123] It will be appreciated that the z-direction carries the most information, followed by one of the dimensions in the x-y plane which may carry more information on the perpendicular movement than the other dimension. For example, the y-axis may be defined as an axis between a user’s head and feet and the x-axis may be defined as an axis from one shoulder to the other. In this example, the y-direction may carry more information on the three-dimensional movement than components of the movement in the x-direction. This is because as the lung inflates, any point on the surface of the lung moves both outwards and upwards, similar to an inflating balloon. Thus, movement in the y-direction is linked to movement along the z-axis.

[0124] These x, y, and z changes can be combined to create a surrogate volume metric, as they are directly proportional to volume changes. For example, the volume change AV may be approximated using:ΔV~ (1 / 2)(4 / 3)π(abΔc + acΔb + bcΔa)where a is the movement in the z-direction, b is the movement in the x-direction and c is the movement in the x-direction. The rear thoracic cavity is fixed in size and thus the volume is halved as shown in the equation. The equation above is a first order approximation which maps chest volume change to an ellipsoid principal axis change. That is, a change in lung volume is taken to correspond to a change in any of the principal axes of an ellipsoid.

[0125] It will be appreciated that while the above techniques for determining a distance to the user may be directly incorporated to determine lung volume and / or changes in lung volume, these techniques may also simply be used to normalise images of users that are taken at different distances to a camera. That is, instead of incorporating distance information directly into lung volume calculations, only x and y-axis information (i.e. a simple 2D image) may be used. Then, only one distance measurement is needed for each frame and this is used to set tracking areas on a user’s chest such that each tracking area always covers the same area on a user’s chest for each user. The distance measurement used for each frame may be taken once only for the first image of the user, or the distance measurement may be taken before image capture commences and may then subsequently be used to normalise all images that are taken of the user’s chest. Alternatively, a distance measurement may be taken for each image taken of the user, using the methods described above.

[0126] Determining a distance between the camera and a user may also mean that it is possible to determine an absolute volume measurement, not just relative volume changes. Because having a distance between user and camera means that it is possible to translate pixel distances captured by the camera into physical distances, an estimate of total physical lung volume may be calculated.

[0127] Two different types of chest movement may be determined. The first is a localised measurement which tracks the movement over time separately for each of the tracking areas and hence separately for each area of the chest covered by a tracking area. The second is a global measurement which is an average measurement of the whole chest overtime that may be calculated by taking an average of the localised measurements of the tracking areas for each image. That is, the global measurement provides information on how the whole of the user’s chest has moved overtime.

[0128] Both of these output measurements can provide valuable insight into a user’s breathing. The global measurement (which may also be termed average displacement) allows determination of a user’s breathing rate and general respiratory function. The localised measurements for the tracking areas enable detailed tracking of a user’s breathing and lung function with respect to different portions of a user’s lung. That is, the localised measurements enable comparison between the breathing pattern that is observed for different parts of the lung. For example, the localised measurements can be compared to determine whether the separate tracking areas are moving synchronously or asynchronously. One or more tracking areas which are moving asynchronously may be indicative of irregularities originating in different parts of the lung, for example one part of a lung not inflating. In a final step S110, information on the determined movement of the user’s chest for the plurality of images comprising the video data is output.

[0129] Figures 6h and 6i show an example plot of tracking movement of a user’s hand placed on the user’s chest and a detrended plot of tracking movement of a user’s hand placed on the user’s chest respectively. That is, Figures 6h and 6i show examples of data obtained when tracking a user’s hand which is placed on the user’s chest in a skin conformal way. As mentioned above, movement of at least one hand is tracked using dedicated libraries, such as mediapipe which enable tracking of different body parts, such as the user’s hand.

[0130] Figure 7 is a flowchart giving more detail on how the grid of tracking areas may be defined and then refined. This may be particularly used for the arrangement shown in Figure 5a in which the tracking areas are equal in size and spaced equally across the user’s chest to form a grid of tracking areas. When tracking a user’s chest, the first step S202 is to determine which area in the first image shows the target area, i.e. the user’s chest. This may be done automatically, e.g. using any suitable technique such as edge detection. Alternatively, guidance may be given to the user to ensure the correct area is tracked. In addition to automatically detecting the target area, it may also be determined at regular intervals if the user is still positioned within the target area. It will be appreciated that some movement of the user unrelated to their breathing may nonetheless be compensated by the tracking methods described below. It will also be appreciated that this first determining step allows for greater freedom in where a user is tracking their chest / breathing movements. For example, movements could be tracked using a mobile camera that is being held by the user. Thus, no predetermined angle or distance to the camera is needed in order for the present techniques to work. The present techniques may compensate for some user and / or camera movement, and as described above, this may also be done by detecting a user’s chest initially and / or periodically while a measurement is taken.

[0131] Figure 8a is a schematic illustration of an application interface on the user device which is used to direct the user to correctly position an image capture device of the apparatus to capture the video data of the user’s chest. A guide outline 604 is shown on the user interface of the apparatus 600. The guide outline can be any useful outline which ensures that the target area 606 (in this case targeting the user’s chest) is captured by the image capture device. In this example, the guide outline 604 shows a user’s head, shoulder and arm outline and when the user positions the apparatus 600 such that the guide outline 604 is aligned with the user’s head and arms, the user’s chest area will be positioned inside the target area 606. In addition to providing the user interface, it may also be monitored if the user has positioned the image capture device correctly to capture a video of the user’s chest. For example, an acoustic, visual, or haptic warning may be issued if the user is no longer positioned correctly in front of the image capture device. When a warning is issued, tracking of a user’s chest movements may be interrupted and only resumed when the user has positioned themselves correctly again. Some user movement may be compensated by the present tracking techniques as described above, so such a warning may only be issued when there is substantial user movement that impedes chest movement tracking. It will be appreciated that such a guide outline can be used to ensure that the markers shown in Figure 5d and / or the pixel areas shown in Figures 5b and 5c are being correctly captured.

[0132] Returning to Figure 7, the tracking areas in the first image may then be defined at step S204 based on the alignment of the target area with the first image. Each tracking area may be positioned such that it covers a subarea of the area showing the user’s chest to obtain as much information as possible on the user’s breathing. As described above in relation to step S104 of Figure 3, an image is typically a grid of pixels and each tracking area (whether using a marker or not), at least initially, may be defined by the location of the outermost pixels.

[0133] The tracking areas defined in the first image may then be saved as template tracking areas S206. The template tracking areas may be used to determine a location of a tracking area in a subsequent image. In a subsequent image, pixel values of the template tracking areas may be compared with pixel values in the subsequent image to determine the movement of a user’s chest.

[0134] Figure 7 describes one method for determining the difference in location of the tracking areas in each image in the video data (i.e. the step of S106 of Figure 3). First, the locations of the tracking areas of a previous image are used to find the tracking areas in a current (i.e. a later or subsequent) image.

[0135] Figures 8b to 8e illustrate how the location of a tracking area is determined from one image of the video to the next to determine the user’s chest movement. Figure 8b shows the location 704 of a tracking area 702 of a first (or previous) image 700. As shown, the location is indicated by the co-ordinates of the top-left pixel in the tracking area. In Figure 8c, the corresponding tracking area 702’ is at the same location 704’ in the next image (i.e. the current image) 700’. In practice, the tracking areas are likely to move from one image to the next, For example due to movement by the user. The area covered by the located tracking areas 702’ may thus merely be an indicative or initial tracking area which is not the best match for the original tracking area. Figure 8d illustrates that in addition to movement in the x and y axes, there can also be movement in the z axis from a first frame to a second frame. As noted above, this z-axis movement can be evaluated using an approximation based on the x and y movement, using a depth capture sensor and / or by using the markers.

[0136] Returning to Figure 7, the next step is to locate search areas based on these initial tracking areas to ensure that there is consistent and accurate mapping from one image to a subsequent image in the video. Such a search area 706 is shown in Figure 8e. Each search area 706 covers the initial tracking area (i.e. the expected location of the tracking area) together with a peripheral area 708 around the initial tracking area. Thus, each search area comprises pixel values at the locations of pixels in and around the corresponding tracking area in the previous image. The search areas 706 may be placed at the same locations in each image for a fixed or predetermined number of images, merely as an example around 30 images. After the fixed number of images have been processed, the locations of the search areas and / or the initial tracking areas may be updated.

[0137] Each peripheral area 708 may be a uniform area around each initial tracking area. In other words, each search area may have an area which is greater than the initial tracking area by a fixed number, e.g. five pixels or any other number, for example, between five to ten pixels or ten to fifty pixels in each direction around each initial tracking area, as is shown in Figure 8e. It will also be appreciated that the number of pixels may depend on the absolute size of an image that is beinganalysed. That is, the absolute pixel size of the search area may be greater for an image of higher resolution, and smaller for an image of lower resolution. Alternatively, the search area may be non-uniform in size and shape.

[0138] Returning to Figure 7, in a next step S210, each search area is searched by defining each subarea which is within the search area. As illustrated in Figure 8f, this subarea 708 has the same size and shape as the template tracking area 710 so that accurate comparisons can be made. At step S212. the pixel values of the pixels within each subarea are compared to the pixel values of the pixels within the corresponding template tracking areas 710 to select a subarea 708 of each search area that most closely matches the corresponding template tracking area. The subarea 708 that most closely matches the template tracking area 710 may be termed the final tracking area and may be determined using any suitable method.

[0139] For example, the final tracking area may be found by assigning a distance score to each subarea within the search area. The distance score may be a measure of similarity between each subarea in the search area and the template tracking area. The distance score may be computed as the intensity difference of each R, G and B channel values between the subarea and the template tracking area. That is, a difference in pixel value for each channel used may be calculated. The subarea with the smallest difference between these pixel values may then be selected as the final tracking area. For example, the difference score may be calculated as:N(|R'ᵢ - Rᵢ| + |G'ᵢ - Gᵢ| + |B'ᵢ - Bᵢ|)i= 0where R, G and B correspond to the red, blue and green colour channels of the template tracking area, R’, G’ and B’ to values of corresponding pixels in each subarea, i is the i-th pixel within the template tracking area or subarea, and N is the total number of pixels within the template tracking area or subarea.

[0140] Additionally or alternatively, the final tracking area may be found by calculating, for each combination of the subarea 708 and template tracking area 710, a cross-correlation value. The crosscorrelation value is a measure of which subarea most closely matches the template tracking area. Hence, the cross-correlation value also is a measure of the most likely location of each final tracking area within the search area. It will be appreciated that selecting an appropriate size and shape for the search area helps to ensure that the final tracking areas which are found are relevant. The comparison of each subarea 708 in the search area 706 may, for example, be achieved by “moving” the subarea 708 across the whole of the search area 706 one pixel location at a time and comparing each of these subareas to the template tracking area 710.

[0141] Returning to Figure 7, the next step S214 is to determine whether there is sufficient confidence in the corresponding tracking areas which have been detected in the current image for subsequent calculations to be meaningful. The tracking areas may be the tracking areas which were located in step S208 or the refined final tracking areas which were output from step S212. When there is confidence that the tracking areas can be used, the method proceeds to further process the current image at step S216. Otherwise the current image is rejected at step S218.

[0142] Confidence in the detected tracking area may be quantified in an appropriate way. For example, the difference score calculated above may be monitored and compared to a typical difference in pixel values. A typical difference in pixel values may, for example, be on the order of 101. If the difference that is detected is on the order of 103instead, the outlier image may be discarded and not processed any further. Alternatively, an average of all previous difference scores may be calculated (i.e. for all final tracking areas in all images before the image currently being processed) to determine a threshold for discarding an image. Alternatively, the average difference score may be calculated from a fixed number of previous images (e.g. five). The difference score of each final tracking area or an average difference score for all of the final tracking areas of the current image may then be compared with the average difference score and the current image may be rejected when the compared difference scores are sufficiently different, e.g. the difference score each final tracking area of the current image or the average difference score is twice as large as the average difference score.

[0143] When there is confidence in the detected tracking areas, a difference in location of each tracking area between the two consecutive images is calculated at step S216. The difference in location may be the distance between the location of a tracking area in the current image and the location of the corresponding tracking area in the earlier image. For example, the difference in location may be calculated by calculating a difference in location between the pixel at the centre of the tracking area of the current and previous images. Alternatively, the difference in location may be calculated using any other suitable method, for example, by comparing the locations of any other pixels, such as the top-left or top-right pixel of the two tracking areas.

[0144] Optionally, the template tracking areas may be updated at step S220. Such an update may be done after each image is processed as described above or more efficiently after multiple images have been processed. This may be done by including information about the tracking areas of the recently processed images in the template tracking areas. For example, each template tracking area may be updated by taking a weighted average of the pixel values for the current saved template tracking area and the pixel values for the corresponding tracking areas in each of the images since the last update was performed. The average may be weighted such that the current template tracking areas are awarded a greater weight than the recently located tracking areas. In this way, if the pixel values of the tracking areas that have just been found deviate largely from the pixel values of the current template tracking areas, this will only have a small impact on the pixel values of the template tracking areas. This will minimise the effect of the update on finding tracking areas in subsequent images. However, updating the template tracking areas can be important in order to ensure that the template tracking areas adapt to changing image conditions, such as, for example, a change in lighting or any other changes.

[0145] At step S222, there is a determination as to whether there are further images in the video data to process. Steps S208 to S220 above are repeated for each image in the plurality of images until the last image is processed. Once all images have been processed, the movement of a user’schest may be determined at step S224. In other words, the method has reached step S108 of Figure 3.

[0146] Figure 9 is a flowchart showing how the detected movement can be transformed into output breathing information. As a brief recap of the data collection steps, an application on a mobile phone may be used to capture video recordings of a user’s face and chest over a minute. After having navigated through the user interface screens (including instructions for use, terms and conditions, guide page), the user clicks on the “Start” button to begin the process. The camera feed starts and is displayed on the screen along with optional guidance for the user on how to position themselves within the designated area. When using the positioning option, once the user’s face is detected by the application and it remains stable within the designated area, the recording automatically starts. Video data may be captured in RGB format and sampled at 60 frames per second for 60 seconds. At the end of this period, the recording automatically stops, data is fed into the pre-processing algorithm and then uploaded to the cloud for further processing,

[0147] The video data captured by the mobile application is typically in a “raw" format comprising RGB values for every pixel on the screen (default resolution: 1280 x 720) per frame (3,600 frames in total). As this amount of data cannot be stored on the cloud within a reasonable timeframe due to its size, an optional pre-processing algorithm may be implemented to downsample the video data and generate a representative breathing signal for each recording. As described above, tracking areas (which may also be termed regions of interest (ROIs)) are defined with the chest area, using the various options described above including pixel defined areas and / or markers. The changes in location of the tracking areas in the x, y and / or z axes are tracked across frames. This displacement value across frames represents the “raw” respiratory signal and is the final output of the pre¬ processing algorithm, which is subsequently stored on the cloud. Thus, in a first step S300, a raw breathing signal is obtained, for example as described above.

[0148] An example breathing signal is plotted in Figure 10a which plots the average movement in the y direction over time wherein the average movement is calculated across all tracking areas in each image which is not rejected. Thus, the breathing signal may also be termed a movement signal. Time is a continuous number from 0 and may be calculated from the number of frames (i.e. images) which are included divided by the number of frames per second. The signal measuring the y-direction movement is typically inverted (i.e. negative values as shown) so that inhalation results in an increase in the signal strength and exhalation results in a decrease. Figure 10a shows examples of individual breaths 502, 504, 506, 508 of a user and the fifth breath is significantly deeper than other breaths.

[0149] Returning to Figure 9, the next step S302 is to apply one or more bandpass filters, e.g. by using appropriate high and / or low band filters such as second order Butterworth filter and / or any other suitable filters. Such band-pass filters may remove low frequency noise (e.g., baseline drifts) and high frequency noise (e.g., due to indoor lighting or motion. The input may be signal such as shown in Figure 10a which covers multiple frequency bands and the output signal covers a reduced set of frequency bands. For example, only frequencies between 0.1 and 2Hz may be let through this way. After applying a filter, the signal may be smoothed as shown at step S304, for example using aSavitzky-Golay filter, or any other suitable filter such as a second order polynomial filter. The inputs to the smoothing may be the filtered signal and a window length (measured in frames per second fps).

[0150] Figure 10b shows a smoothed signal derived from the raw movement signal in Figure 8a and thus the same symbols are used. The first few individual breaths are 502’, 504’, 506’, 508’. In addition to being smoothed to reduce noise in the signal, the values of the y movement are shifted so that not all values are negative. The zero value is set at a value which is approximately the median or average of the values of the peaks and troughs. Temporally smoothed the filtered signal may help to further reduce noise and facilitate later steps such as breathing cycle detection. The signal shown in Figure 10b may be termed the “processed signal”.

[0151] Returning to Figure 9, the next step S306 is to calculate a gradient (or first derivative) of the filtered, smoothed signal. The inputs to the smoothing may be the smoothed signal, a window length (measured in frames per second fps - this may be a 1 second window) and the details of the method used. There are various options for calculating the gradient, including use a median filter, specifying how many frames to use and a specifying a blanking time (i.e. minimum distance between detected peaks). An example process may be to invert the input signal, e.g. by subtracting it from 255, then applying a three-frame median filter and finally computing the gradient for each frame (i.e. each image) by using a least squares fit for the frames data. The output is the gradient signal and the first few frames will be zero.

[0152] Returning to Figure 9, the next step S308 is to apply temporal smoothing to the gradient of the breathing signal, for example by applying a Savitzky-Golay filter as described above. Figure 10c shows a gradient of the smoothed breathing signal in Figure 10b. The first few individual breaths are 502”, 504”, 506”, 508”. Stationary points are indicated by red star symbols, peaks by orange dot symbols and troughs by green triangle symbols. The stationary points are the peaks and troughs in the gradient signal.

[0153] Returning to Figure 9, a next step S310 is to identify breathing cycles, for example using a breathing cycle detection algorithm to identify the start and end of inhalation and exhalation cycles. Any suitable algorithm may be applied, for example by using the stationary and zero crossing points of the gradient signal and the breathing signal. Each of the peaks, troughs and zero crossing points may be labelled with an index which indicates their location in the gradient signal. An output from this step may be an array of frame indices indicating the location of the peaks, troughs and zero crossing points in the gradient signal which may be termed the “gradjndices”, an array of frame indices indicating the location of the peaks, troughs and zero crossing points in the original signal which may be termed the “rawjndices”, and an array of the labels for each index which may be termed “labels”. The pattern of stationary and zero crossing points found in the gradient signal and in the breathing signal may be used to reject any implausible results. For example, when there are two adjacent zero crossing points in the gradient signal, the corresponding stationary points in the breathing signal are analysed. The output from this step will typically be a pattern of stationary and zero crossing points which follow the normal pattern, e.g. zero to peak to zero to trough to zero. The pattern of stationaryand zero crossing points may then be analysed to detect or isolate “complete” breaths within the breathing signal.

[0154] As shown at step S312, there is then a determination as to whether the quality of the signal is of sufficient quality. The quality determination may be done by a visual inspection of the gradient and / or breathing signals. The quality determination may be automated by isolating the signal of each breathing cycle and aligned each breathing cycle relative to their peaks. A breath template may then be generated by averaging the individual isolated breaths. Each breath may then be then scored according to their similarity to the breath template. A similarity score may be calculated across all breaths for an overall signal and compared to a predefined threshold.

[0155] If the quality threshold is not met, a signal is labelled as “bad” or rejected at step S314. Alternatively, if the quality threshold is met, the signal may be labelled “good quality” and the process can continue. It will be appreciated that by using more than one threshold, additional classifications may be obtained, e.g, “very good”, “good" and “bad.

[0156] For the signals which are deemed usable (e.g. at least “good”), based on the detected breaths, a user’s respiratory rate may then be calculated at step S318. The respiratory rate simply is the number of breaths in a minute. The respiratory rate may thus be calculated by counting how many breath cycles have been detected and by computing the total duration as the time of the first detected breath cycle to the end of the last detected breath cycle. The total duration is then divided by the number of breath cycles to obtain the respiratory rate. The final step S320 is to generate the desired outputs from the information calculated above.

[0157] Examples of such outputs are shown in Figures 11a to 11 f which are loop diagrams also known as displacement loops, corresponding to individual breaths of the breathing signal of a user. These examples have been determined using the approach having the markers as shown in Figure 5d. The x-axis in a loop diagram represents time and the y-axis represents velocity. An inhalation period (period between minimum and maximum of breathing signal) of an individual breath is plotted on the negative side of the y-axis and the smoothed gradient breathing signal is inverted as necessary, so that it has negative values. The inhalation period is plotted in negative time direction, i.e. such that the last point of the inhalation period is plotted at the origin. An exhalation period (period between maximum and minimum of breathing signal) of an individual breath is plotted on the positive side of the y-axis and the smoothed gradient signal is inverted as necessary, so that it has positive values. The loop diagrams of Figures 11a to 11f are equivalent to loop diagrams obtained using conventional spirometry such as the loop diagram of Figure 1a to 1f. In the diagrams shown in Figures 11a to 11f the x-axis shows volume (Z, m) and the y-axis shows flow (dZ / dFrame). These quantities are equivalent to conventional flow loops as described above.

[0158] Different sectors of the loop diagrams may be indicators for different aspects of lung function and thus the loop diagrams may be analysed to extract information about each breath. For example, the loop diagram may be divided into quadrants to analyse lung function. The duration (time) between consecutive fiducial points (i.e. consecutive stationary or zero crossing points - zero to peak, peak to zero etc) may be calculated. The volume (V1, V2, V3, V4) between consecutivefiducial points may additionally or alternatively be calculated as the trapezoidal area between the gradient and time axes. Additionally, or alternatively, the shape of the positive section of the loop diagram, close to the origin may provide an indication of the severity of COPD.

[0159] While taking a video of themselves breathing, the user may be instructed to perform specific breathing exercises or breathe in a specific pattern to obtain a loop diagram that is of maximum diagnostic use. For example, the above mentioned methods may be used to obtain the Force Expiratory Volume (FEV1) which can indicate how severe a patient’s chronic obstructive pulmonary disease (COPD) is. FEV1 may be obtained by instructing the patient to perform maximum voluntary ventilation (i.e. a patient breathing as deeply and quickly as possible for about 12 seconds). Similarly, the above techniques may be used to obtain the Force Volume Capacity (FEC). Additionally, having a patient perform a set of breathing manoeuvres may help obtain accurate data, as there is less movement that is not due to breathing. Hence, the problem of having to filter out nonbreathing related movement can be minimised.

[0160] Any of the above-mentioned metrics may be used as indicators of lung function. Lung function may be expressed using any suitable metric, including at least one or more of the following:• Respiratory rate: number of breaths per minute.• Respiratory pattern (e.g. over a fixed time frame such as a minute),• Forced expiratory volume (measured as FEV1, FEV2, FEV3, FEV4, FEV5 and / or FEV6):forced expiratory volume is how much air a person can exhale during a forced breath after 1, 2, 3, etc. seconds,• Forced vital capacity (FVC): forced vital capacity is the total amount of air exhaled during a fast spirometry manoeuvre,• Vital capacity (VC): vital capacity is the total amount of air exhaled during a slow spirometry manoeuvre,• PEFR (peak expiratory flow rate),• Tidal volume,• Minute ventilation: volume ventilated over a minute,• FEF (forced expiratory flow) at 25%, 50% and 75%,• Signal quality: Template matching score (i.e. breath self-similarity),• Properties of a displacement loop as a surrogate of a flow-volume loop,• Displacement loop volume as a surrogate of a flow-volume loop volume, e.g. twice an expiratory volume and / or twice an inspiratory volume, and / or• Displacement loop times, e.g. twice an expiratory time and / or twice an inspiratory time,

[0161] Lung function may further be expressed as any suitable ratio of the above quantities, for example, as a ratio of FEV1 / VC. Lung function may, for example, be assessed by taking the best metrics calculated from a number of measurement attempts. For example, lung function may be assessed by taking the best of three measurements. The best value may be the highest or lowest value achieved, depending on the measured quantity. For forced vital capacity, vital capacity and forced expiratory volume, the value may be better the higher the measured quantity.

[0162] To assess lung capacity, these values may be analysed by, for example, obtaining a reference value for the user. The reference value may represent an average expected measurement for a comparable user. The reference value may be obtained by using the user’s height, age and / or sex. BMI and / or smoking status may further be considered. Measured values for any of the above quantities may then be compared to the reference values, for example, by calculating a percentage value of the measured value to the reference value. The reference value may comprise an average value and / or a standard deviation for that average value. Additionally, the reference value may comprise a value for a lower limit of a normal range. Measured values may then be compared to the reference value for a lower limit of the normal range. The reference values may further be used to calculate a standardised residual (Z-score). This may be used to gauge a strength of difference between the measured and reference values. A normal value for the standardised residual may, for example, be between -1.64 and 1.64.

[0163] Template matching may, for example, be used to ascertain if displacement caused by a breath results in the same shape of recorded breath. Breaths of irregular shape may indicate, for example, decreased lung function and / or other lung disease. Sometimes, breaths of irregular shape may also indicate suboptimal signal quality. Consequently, in response to a low template matching score, the user may be prompted to adjust the images of their chest that they are recording. For example, the user may be prompted to move to a place with better lighting conditions. Additionally, or alternatively, other prompts to improve signal quality may be presented to the user. For example, the image of a displacement loop may be divided into smaller sections, and each smaller section may be matched against a template image. Template matching may be performed using any suitable technique, such as, for example, neural networks and / or deep learning techniques, such as Convolutional Neural Networks (CNNs). Additionally, or alternatively, template matching may be performed after an image of the displacement loop diagram has been processed by a pyramid filter to reduce the image’s resolution while preserving relevant features. Any suitable template matching algorithm may be used, such as algorithms available in publicly accessible libraries, such as OpenCV.

[0164] Other metrics to assess lung function may include a classification as to how likely it is that a user has chronic obstructive pulmonary disease (COPD) and / or asthma, including exercise induced asthma or a stratification of a user into a first user group which has COPD and / or asthma or a second group which is healthy. This probability classification or stratification may, for example, be assessed using the above metrics. For example, airway obstruction may be indicated by a reduced ratio of FEV1 / FVC and / or FEV1 / VC compared to a reference value. The reference value for a ratio of FEV1 / FVC may, for example, be 0.7, although this value may vary based on age and / or gender.

[0165] Other measures that may be used are, for example, be breathing variability, i.e. if there are any changes in a user’s breathing rate. This may, for example, be evaluated by measuring the time between a user’s breaths. And average time between breaths and standard deviation of this value may be calculated. A large standard deviation may point to an irregular breathing rhythm. This may be an indicator of a likelihood of respiratory disease. Other measures for evaluating breathing variability may be used, such as calculating a breathing rate for a plurality of time windows of themeasured breathing signal. The plurality of time windows may, for example, be calculated by shifting a time window of predetermined length along the breathing signal. In such a manner, average breathing rates for the duration of the breathing signal may be found. An average breathing rate and corresponding standard deviation, or other measure of breathing variability may be calculated.

[0166] Lung function may further be assessed by determining a user’s recovery rate. Recovery rate may mean the time it takes a user to recover from physical activity. For example, this may mean the time it takes for a user’s respiratory measurements to return to their resting values after a period of physical exertion. To determine recovery rate, a user may be asked to record a video of their chest while resting. The user may then be asked to perform some physical activity, such as, for example, walking a short distance. The user may continue recording their chest movements during the period. The user may then be asked to rest again while continuing to record the video of their chest. Metrics of lung function may be calculated for the whole length of the video recording. Metrics recorded at the beginning of the recording, when the user is resting, may be used as threshold values that have to be recorded again after the user has exercised. When the threshold values are measured again, the user may be considered as having recovered from the exercise. The time elapsed between recording the start of the exercise, and the measured values returning to the threshold value may be considered the recovery rate / time.

[0167] Additionally or alternatively, lung function may also be assessed by calculating a ratio of area under the curve for an inhalation and exhalation, i.e.:Area ExhalationArea InhalationFor example, when looking at Figures 1a to 1d, it is clear that not only the shape of a flow loop diagram, but also an area below the curve changes with an obstructive breathing pattern. Area above the x-axis in Figures 1a to 1d shows exhalation, area below the x-axis shows inhalation. However, it will be appreciated that both of these areas may be referred to as “areas below / under the curve”, as plotting a curve above or below the x-axis is simply convention and could easily be switched around. Figure 1c in particular shows an obstructive breathing pattern. For patients with obstructive pulmonary disease, area under the curve during inhalation is often significantly greater than during exhalation. Thus, the above ratio is smaller for users with an obstructive lung disease than for healthy users. For example, the above ratio may be 0.64 for a user with obstructive lung disease, whereas it may be 0.97 (i.e. close to 1) for a healthy user with no pulmonary issues. Thus, a classification of how likely a user is to have an obstructive pulmonary disorder, such as COPD, may involve calculating the above mentioned ratio of area under the curve for an inhalation and exhalation and comparing this to a reference ratio. For example, a reference ratio may be 0.8, 0.85 or 0.9 and when a ratio calculated for a user is below this reference ratio, the user may be classified as having an obstructive pulmonary disorder. It will also be appreciated that the above ratio may be calculated for several loop diagrams of the user and then averaged. For example, a median, arithmetic average or any other suitable averaging method may be used to calculate an average when more than one ratio is calculated forthe same user. When the above ratio is output as a measure of lung health, the averaged ratio may be output.

[0168] These lung measures typically vary over time. The methods described herein are particularly suitable for recording lung function of a user over extended time intervals, as they may be performed by a user at home using, for example, their mobile phone. Such long term data may be particularly useful for long term monitoring of a user’s lung health. Long term measurement may also be useful to establish a user’s baseline for certain measures of lung health. The user and / or a medical professional may then be alerted when the user’s lung health suddenly declines compared to what is normal for the user. For example, when declining lung health is detected, a user may be sent an alert message. This may be particularly useful when, for example, the user is an athlete that suffers from exercise induced asthma. This is a common type of asthma amongst professional athletes. The athlete may measure their lung health while exercising and may be alerted if their lung health is declining. This may indicate to the athlete that they should pause exercising to ensure that an asthma attack does not occur.

[0169] Monitoring different metrics of lung health over time also means that it can be determined how consistent a certain measure of lung health remains over time. As described above, this may be useful to determine how lung health varies overtime for a user.

[0170] Additionally, the user’s face may also be detected, and the recording used to obtain a value for the user’s heart rate. To detect changes in blood volume in a user’s face, it is important to choose an area of the face that allows for easy monitoring of blood flow, i.e. that is transparent enough for blood flow to be visible in this area. This can be done by detecting blood volume changes in, for example, the cheek and forehead areas.

[0171] The diagrams shown in the above mentioned figures may be processed using a machine learning, ML, or Artificial Intelligence, Al, algorithm. These algorithms may be used to perform an analysis of a patient’s breathing pattern and evaluate their lung function.

[0172] Figures 12a and 12b combined are a flowchart showing an example method for training an ML algorithm. The data used to train the ML algorithm is obtained from a first clinical study conducted in Japan and ten smaller clinical studies conducted in the UK. Participants were screened and selected either by the external agency or their primary care physicians. The full cohort involved patients with a history of COPD, individuals with potentially undiagnosed CORD, patients suffering from ILD, as well as healthy participants with no recent surgeries, no medical conditions (including high blood pressure, angina, recent heart attack), nor use of specific medication (including beta blockers, beta agonists and steroids). While the minimum age required to enrol in the studies was set to 18, all participants ultimately recruited were above 21. The main demographic information of the full cohort is shown in the table below.Sex Age range Ethnicity Nr of patients per conditionsMale 21-80 years of age 49 Japanese, 20 White 51 normal, 23 obstructive, 14British, 17 Mixed British, 2 restrictiveUnknownFemale 22-78 years of age 49 Japanese, 22 White 48 normal, 27 obstructive, 12British, 13 Mixed British, 1 restrictiveIndian, 1 Black Caribbean,1 Unknown

[0173] All the cohort wase tested on a medically-graded spirometer device (Vitalograph Pneumotrac SP6 In UK; Nihon Kohden, Model Number HI-302U in Japan) which was calibrated at the start of the day prior to use. Participants were instructed and coached throughout each spirometry manoeuvre by a health-care assistant, completing a set of up to 3 fast spirometry (FS) blows (i.e. deep inhalation followed by a forced fast exhalation).

[0174] After spirometry, participants were tested with the mobile application (rCP) which is described above. They completed up to 3 tidal breathing (TB) recordings and up to 3 fast spirometry (FS) recordings with the same mobile application. During TB recordings, participants were instructed to breathe naturally in an upright position facing the device while remaining as still as possible. During FS recordings, participants were instructed to start breathing normally and, when signalled by the healthcare assistant (usually within 10-20s from the start of the recording), completed a fast spirometry blow with as minimal upper body movement as possible. If there was still time in the 60s recording at the end of the manoeuvre, participants were asked to return to a normal breathing pattern.

[0175] For participants recruited via the UK clinics, clinical labels were assigned to each patient based on the medical records provided by their primary care physicians. These records included a diagnosis of obstructive or restrictive disease along with the cause and the test that led to the diagnosis, in the absence of any of the two from the medical records, the patient was labelled as “normal” (i.e. healthy). In the Japanese cohort, “ventilation impairment” labels were automatically assigned during data collection by a medically-graded spirometer to each of the fast spirometry tests performed (3 per participant). This information was then used to define a participant-specific label as follows: A participant is tagged as “normal” if they are labelled “normal” in any of the spirometry repetitions for “ventilation impairment”. This means the best-performing spirometry result is used to label the individual. If none of the repetitions label them as “normal,” the individual is labelled “obstructive” or “restrictive” if either of these terms appears in all of the “ventilation impairment” entries. There were no instances where “obstructive” and “restrictive” labels appeared on the same individual.

[0176] There are two sources of information that have been used in model development. The first consists of signals extracted via the mobile app, associated to either TB or FS manoeuvres, which include processed signals and gradient signals. As described above, the processed signal is the signal shown in Figure 10b in which the raw signal from the application has been filtered and temporally smoothed. The gradient signal is the gradient (or first derivative) of the filtered, smoothed signal. The second source is the metadata corresponding to the participants from whom the signals were recorded. This metadata includes the various demographic features of the recruited individuals and clinical features related to their respiratory conditions: normal, restrictive, and obstructive.

[0177] As shown in Figure 12a, the first step S1202 is to merge the two sources of information so that each recorded signal was meticulously labelled with the corresponding metadata of the individual. As shown in the bar charts of Figures 13a to 13c, the merged database comprises 175 individuals with an almost equal male-to-female ratio of 88 to 87 (Figure 13b). Notably, as shown in Figure 13a, there is a 1:2:4 ratio of individuals with respiratory restrictive conditions to those with obstructive restrictive conditions to those with normal respiratory conditions. It is also important to note that not all individuals have signal recordings for all manoeuvres, with a 1:2:6 ratio of individuals having only TB signals, only FS signals, and both TB and FS signals, respectively as shown in Figure 13c. Additionally, Figure 13d shows that most individuals have 2 TB signals and 3 FS signals recorded, resulting in a database with a total of 876 signals for both manoeuvres.

[0178] Returning to Figure 12, to ensure an appropriate data split for training and testing (maintaining consistent ratios across gender, available manoeuvres per individual, and importantly, stratifying with respect to the respiratory clinical labels, while avoiding data leakage), after collecting the full set of recordings and labelling the patients using their metadata, the next step is to create a unique identifier for each patient. The unique identifier may be calculated in any way, e.g. by concatenating their gender, available manoeuvres, and clinical label. At step S1206, the labelled data is then stratified into training data and verification / testing data. For example, the data may be split 80 / 20 based on this identifier. This approach mitigates data leakage by ensuring that all signals associated with a given individual are assigned exclusively to either the training or testing set, thereby facilitating model comparisons.

[0179] Once the training and testing data has been defined, the method can proceed with an automated development pipeline for training and testing the various models of interest. At step S1208, there is a selection of the classifier to be trained. The selected classifier is preferably a binary classifier and may be selected from normal vs abnormal, restrictive vs no-restrictive, restrictive vs obstructive and obstructive vs normal, with particular interest in the binary classifications of normal vs abnormal and restrictive vs no-restrictive. The normal vs abnormal classification is significant as it could facilitate the use of respiratory signals as a screening tool to evaluate the likelihood of an individual having a normal respiratory profile. The restrictive vs no-restrictive classification is useful in the prevention of restrictive respiratory conditions. To develop a robust multiclass classifier for normal v. restrictive vs obstructive categories, the aforementioned binary classifiers may be complemented with additional binary classifications: restrictive vs obstructive and obstructive vs normal. This approach allows the complex multiclass problem of normal vs. restrictive vs. obstructive classification to be decomposed into more manageable and analytically tractable binary problems, improving overall model performance and interpretability.

[0180] It is noted that initial attempts at solving the multiclass classification problem yielded balanced accuracies consistently below 0.70, highlighting the complexity and difficulty of addressing this problem with a single, unified model. The results below for the binary classifiers show better accuracy. In addition to the inherent technical complexity on the modelling side (especially due to the amount of data points available and class imbalance), there might also be a clinical componentinvolved in the difference in model performance. Specifically, there is a clear clinical correlation where subjects may have a clinically diagnosed ILD that does not exhibit a sufficient level of restrictive pattern factors to fit into a specific class. Furthermore, the proportion of restrictive cases may include features common to both normal and obstructive patterns of disease.

[0181] The binary classification is also useful for investigating various potential discriminative issues, leveraging the current signals to achieve clear distinctions. These binary classification problems can then be strategically used in a hierarchical approach to effectively resolve the multiclass classification problem. In other words, these models will subsequently be integrated to construct a hierarchical classifier for multiclass classification.

[0182] At step S1208, the binary classification problem is currently selected from the following options: normal vs. abnormal, restrictive vs. no-restrictive, normal vs. obstructive, and obstructive vs. restrictive. For the first two scenarios, the entire dataset of recordings is used, while for the latter two, the restrictive and normal subpopulations are excluded, respectively. Accordingly, at step S1210, there may also be a selection of the signals from the training dataset which are to be used. The selection may be based on the type of the signal and / or the manoeuvres which were used to generate the signals. Thus, there is a selection of TB or FS and gradient v processed signal. Step S1210 may be performed in parallel with step S1208 or they may be performed in any order.

[0183] Once the initial selections have been made at step S1212, there is an optional preprocessing step. For example, mean-variance scaling may be applied using any suitable technique, e.g. using the tslearn preprocessing described in “Tslearn, a machine learning toolkit for time series data” by Tavernard et al published in The Journal of Machine Learning Research in 2020.

[0184] The next steps use various shapelet-based transformations on the set of signals extracted from the previous step. The shapelet approach was proposed by “Time series shapelets: a new primitive for data mining “ by Ye and Keogh published in 2009 in Proceedings of the 15th ACM SIGKDD. The shapelet approach is a powerful and interpretable method for time series classification that focuses on identifying and utilising local patterns, or "shapelets," within the data. Shapelets are sub-sequences of time series that are highly discriminative for classifying the series into different categories. Unlike traditional methods that may rely on global features or complex transformations, shapelet-based techniques directly search for and leverage these local patterns, making them particularly effective and understandable. To determine the presence of a shapelet, the distance between the predefined shapelet and all same-sized subsets of the time series is commonly calculated. If this distance is sufficiently small, the shapelet is considered to be present in the time series.

[0185] Figure 14a plots value against time for a time series and a shapelet overlay. The distance between a time series (blue) and a shapelet (red) is determined by sliding the shorter shapelet along the length of the time series. Figure 14b plots the Euclidean distance against time and shows the point-wise distances at each position (green). The smallest distance encountered during this process is returned.

[0186] In a classification setting, the primary objective is to identify the most discriminant shapelets within labelled time series data. These shapelets serve as essential features that capture the distinguishing characteristics of different classes or categories present in the data. There are two primary approaches to obtaining shapelets: mining from the training set or learning through gradient descent. In the mining approach, shapelets are extracted directly from the training data, seeking subsequences that exhibit high discriminative power between different classes. This process involves exhaustive search algorithms or heuristic methods to efficiently explore the space of potential shapelets. On the other hand, the learning approach involves iteratively adjusting the parameters of candidate shapelets using gradient descent or other optimization techniques to minimise classification error. This method enables the discovery of shapelets that are tailored to the specific classification task at hand, potentially leading to more effective discrimination between classes. There are other techniques for generating shapelets such as rotation-invariant similarity measures and bag-of-patterns representations, dynamic time warping (DTW), the Fast Shapelets algorithm and a scalable algorithm for discovering shapelets using a divide-and-conquer strategy.

[0187] One particular implementation which is used is Grabocka's algorithm which is described in " Learning time-series shapelets" by Grabocka et al published in " Proceedings of the 20th ACM SIGKDD” in 2014 and is incorporated by reference. Grabocka's algorithm is a shapelet discovery algorithm that employs a heuristic gradient descent search procedure. This approach, referred to as Learning Shapelets (LS), identifies, at step S1214 a predefined number of shapelets that are not confined to being subseries within the training data. The set of shapelets may be initialised through a K-means clustering of candidate sub-sequences from the training data. At step S1216, there is an optimisation of the parameters of the shapelets. Among the various hyperparameters of the shapelets that can be tuned, different numbers of shapelets and sizes of shapelets can be used as adjustable parameters. These parameters were explored within the range of [10, 100], as they are critical for capturing the most relevant patterns in the signal data: The number of shapelets determines how many distinct patterns are to be identified in the signals. The size of the shapelets controls the length of these patterns, impacting the granularity of the features extracted.

[0188] The objective function for the optimization process is a logistic loss function, which includes a regularisation term, and is based on a logistic regression model for each class. The algorithm jointly learns the weights for the regression and the shapelets in a two-stage iterative process, ultimately producing a final logistic regression model. At step S1220, there is a check to see if more iterations are required to train the shapelet parameters and the weights for the regression model. If so, there is a loop back to the identifying step and thus another iteration is carried out. For all cases, a sufficiently large number of iterations of the optimizer was used, for example 500.

[0189] The scalability and performance of Grabocka's algorithm make it a popular method for shapelet-based classification tasks. In general, shapelet-based approaches to time series classification, stands out for its unique blend of interpretability and generalizability, distinguishing it from more opaque deep learning methods. By transforming time series data into a new feature space based on discriminative subsequences, or shapelets, this method enables the application of standardmachine learning models, such as logistic regression, support vector classifiers or decision trees, over the transformed distance space. This inherent interpretability allows for a transparent understanding of the classification process, as shapelets represent human-interpretable patterns that can be visualised and analysed. Moreover, shapelet-based models tend to offer greater generalizability than deep learning approaches, particularly when dealing with limited data, as they leverage the discriminative power of shapelets to capture essential patterns effectively. This makes shapelet-based classification a robust and practical strategy for problems where data scarcity is a significant concern, such as in the one proposed in this application.

[0190] At step S1220, there is an optional step S1220 of selecting the most significant features for the optimised shapelets. For example, the Mann-Whitney U-test may be applied to select the most significant features (distances to shapelets) for distinguishing the two classes defined in step S1208. For example features with a p-value < 0.05 may be retained. To address potential class imbalance issues, there also an option (not shown) of evaluating using oversampling techniques like SMOTE which is described in “SMOTE: synthetic minority over-sampling technique by Chawla et al published in Journal of artificial intelligence research in 2002.

[0191] The optimised distance-based transformations of the signals obtained from step S1216 are used to train at step S1222 a variety of machine learning classifiers on this lower-dimensional representation. Given the shapelets' ability to linearly distinguish between classes, the focus is on linear-based classifiers, including support vector classifiers (SVC) and K-nearest neighbours (KNN), in addition to the logistic regression model integrated in Grabocka's algorithm. The binary classifiers above are trained using a grid search cross-validation approach over a repeated and stratified K-fold strategy (3 repetitions and 3 folds). For SVC, we tune hyperparameters such as the regularisation parameter (ranging from 0.0001 to 1000) and the kernel type (linear, sigmoid and rbf). For KNN, we tune the number of neighbours (ranging from 1 to 10) and the Minkowski distance parameter (1 or 2). Additionally, we explore more complex tree-based models, including XGBoost, Extra-Tree, Random Forest, and LightGBM, using the AutoML library which is described in “Flaml: A fast and lightweight automl library” by Wang et al published in Proceedings of Machine Learning and Systems in 2023. Given the nature of these models, which are primarily boosting and tree-based, we incorporate a popular recursive and SULOV-based feature selection strategy via the featurewiz library [Seshadri2020] among the preprocessing options.

[0192] At step S1224, the trained model is saved, e.g. as a.pkl file. The model is then tested at step S1226 using the portion of the test data (e.g. the 20% of data which is reserved for testing). As step S1228, there is a determination as to whether more binary classifiers need to be generated and if so, the process loops back to step S1208 in Figure 12a and the whole process is iterated.

[0193] Otherwise, after training and testing all binary models, various performance metrics are computed at step S1230. Example metrics including some or all of balanced accuracy, PRAUC (Precision-Recall Area Under the Curve), average precision, and AUROC (Area Under the Receiver Operating Characteristic Curve). At step S1232, the top-performing models are selected, e.g. using a Pareto selection approach. For example the top models may be those which lie on the Pareto frontaccording to the performance metrics. The models with the highest balanced accuracy are prioritised due to the imbalanced nature of the problem, ensuring its competitive performance is also observed during training.

[0194] For example, the table below shows the performance metrics of models exhibiting the highest balanced accuracy -per binary problem- when evaluated on the test signal batch. There are four binary classification problems: restrictive versus no-restrictive, normal versus abnormal, obstructive versus normal, and obstructive versus restrictive. These classifications were based on input data obtained from either FS or TB manoeuvres as either gradient or processed signal:Binary Problem Restrictive - No Normal - Obstructive - Obstructive - Restrictive Abnormal Normal Restrictive Manoeuvre TB FS FS FSSignal Type Processed Gradient Gradient GradientMethod SVC (C=215.44, SVC (C=1000, SVC (C=1000, Extra-Tree (#class weight = class weight = class weight = estimators = 27, None, kernel = None, kernel = None, kernel = max features = rbf) rbf) rbf) 0.21, max leaves = 10, criterion = gini)Feature Mann-Whitney None / SMOTE None / SMOTE Featurewiz / Selection / U-test / SMOTE SMOTE OversamplingShapelet Shapelets Shapelets Shapelets Shapelets Configuration length / amount: length / amount: length / amount: length / amount:50 / 50 100 / 100 100 / 50 50 / 100 Performance Balanced Balanced Balanced Balanced accuracy: 0.89, accuracy: 0.78, accuracy: 0.79, accuracy: 0.82, AUROC: 0.90, AUROC: 0.80, AUROC: 0.84, AUROC: 0.84, PRAUC: 0.56, PRAUC: 0.79, PRAUC: 0.69, PRAUC: 0.73, Average Average Average AveragePrecision: 0.59 Precision: 0.80 Precision: 0.71 Precision: 0.74

[0195] The data in the table serves as a comprehensive display of the competitive performance achieved by a diverse array of methodologies across each meticulously examined binary classification problem. This presentation underscores their potential as robust and versatile tools applicable to a wide range of scenarios, prominently including the discernment between individuals presenting abnormal (obstructive / restrictive) breathing patterns and those manifesting normal respiratory function. Notably, among these binary classifications, the task of distinguishing restrictive from non-restrictive breathing patterns emerged as particularly beneficial, demonstrating strong performance despite encountering substantial class imbalance within the dataset.

[0196] At step S1234, the best binary classifiers may be combined or merged to generate a multi-classifier. In particular, the classifiers may be combined to address the multiclass problem of normal vs obstructive vs restrictive by combining two binary models in a hierarchical fashion. From now on, the first model in the hierarchy is referred to as the filter model and the second one as thereduced model. Specifically, combinations of two specific hierarchical ensembles and two manoeuvre alternatives are considered.

[0197] The two hierarchical ensembles are: Hierarchy 1 - first considering the Restrictive vs No Restrictive problem (as filter model) followed by Normal vs Obstructive (reduced model) for the cases predicting in the first step as No Restrictive; Hierarchy 2 - considering the Normal vs Abnormal classification problem (as filter) and then, for the cases predicted as Abnormal, the binary classification Restrictive vs Obstructive (as reduced model). For the manoeuvres, there are the following options: Single manoeuvre in both filter and reduced models (i.e., either TB / TB or FS / FS) and Dual manoeuvre, considering signals coming from different manoeuvres in the two models of the hierarchy (i.e., either TB / FS or FS / TB).

[0198] While in the single-manoeuvre alternative the same signals will be used in the evaluation of both binary models in the hierarchy, dealing with the dual-manoeuvre alternative is not so straightforward. In fact, as there are often multiple TB and FS signals for the same individual (and not necessarily in the same amount for both manoeuvres), the signals need to be suitably combined for the hierarchical problem. The process may thus involve generating as many signal pairs as the number of combinations of recordings associated with the manoeuvre of the filter model and recordings associated with the manoeuvre of the decision model. In other words, if for patient X we have 2 TB signals and 3 FS signals and the dual-manoeuvre alternative is considered, then we generate 6 signal pairs (1st TB with 1st FS, 1st TB with 2nd FS, etc.) for evaluation of the hierarchical problem. Note that for some cases there are no recordings associated with a specific manoeuvre. These cases will always be neglected when signals from that particular manoeuvre are required for the evaluation of the filter model. However, in the second step of the hierarchical model, having missing signals is problematic only for the cases in which evaluation of the reduced model is actually required and hence the considered case will be discarded in this scenario.

[0199] The table below presents the most relevant test set outcomes out of the over 700 hierarchical approaches tested in this section, corresponding to the highest balanced accuracies achieved with the hierarchical approach using the manoeuvres TB / TB, FS / FS, and TB / FS or FS / TB for the filter and reduced models, respectively. It is important to note that for this task, we evaluated all models in the Pareto front for the various binary classifiers.Manoeuvre TB / TB FS / FS Mixed (FS / TB or Combination TB / FS)Filter model Restrictive-No Normal-Abnormal Restrictive-No Restrictive (SVC over (SVC over gradient Restrictive (SVC over processed signal of signal of FS processed signal of TB manoeuvre) manoeuvre) TB manoeuvre) Reduced model Normal-Obstructive Obstructive- Normal-Obstructive (SVC over processed Restrictive (Extra-Tree over gradient signal of TB Tree over gradient signal of FS manoeuvre) signal of FS manoeuvre)manoeuvre)Performance Balanced Accuracy: Balanced Accuracy: Balanced Accuracy:0.742 (34 / 49 samples 0.711 (66 / 92 samples 0.754 (94 / 129 predicted correctly). predicted correctly). samples predicted Per class: 18 / 28, 7 / 12 Per class: 42 / 51, correctly). Per class: and 9 / 9 normal, 13 / 28 and 11 / 13 69 / 82, 16 / 38 and 9 / 9 obstructive and normal, obstructive normal, obstructive restrictive signals and restrictive signals and restrictive signalspredicted correctly. predicted correctly. predicted correctly.

[0200] The proposed options in the table above outperform the previously mentioned alternatives. Specifically, for the mixed manoeuvre combination (FS / TB), the balanced accuracy reached a balanced accuracy of 0.754, which is particularly competitive when compared to the previously evaluated ML / DL multiclass classifiers discussed above (for which this metric remained below 0.670 for all explored alternatives). In this case the hierarchy classification was composed by an initial Restrictive-No Restrictive classifier (using TB signals) and a Normal-Obstructive classifier (using FS signals). Moreover, this approach demonstrated particularly strong performance in identifying restrictive respiratory cases, effectively identifying all positive cases.

[0201] In summary, the pipeline can be summarised in five steps: firstly the identification of the classification problem and the type of signal being analysed; secondly the selection of the signal preprocessing technique, if any; thirdly the configuration and application of shapelet transformation; fourthly the selection, training, and validation of machine learning models using the distance-based shapelet-transformed signals; and finally, testing and selection of the best-performing.

[0202] Once the binary classifiers are selected and optionally combined into a multi-classifier, the classifiers can then be used to assess patients. Figure 15 shows an example flowchart with the steps of classifying a patient. In a first step S1500, a signal from a patient to be classified is received. It will be appreciated that a single signal is considered for ease of description but there may be multiple signals which are processed. In step S1502, there is a determination as to the nature of the manoeuvre which the patient performed to generate the signal. Depending on the determination at step S1502, a first binary classifier (also termed filter model) is selected at step S1504. For example, when the user has performed tidal breathing (TB), the first binary classifier may be a classifier which classifies a user as restrictive vs no-restrictive. Alternatively, when the user has performed fast spirometry (FS), the first binary classifier may be a classifier which classifies a user as normal vs abnormal. At step S1506, there is also a selection of a second binary classifier (also termed a reduced model) and this selected may be based on the determination at step S1502 and / or the selection of the first binary classifier. For example, when the restrictive vs no-restrictive binary classifier is used, the second binary classifier may be Normal-Obstructive and when the normal vs abnormal is used, the second binary classifier may be Obstructive-Restrictive. It will be appreciated that by appropriate selection of the combination of the first and second binary classifiers, it is possible to classify patients as normal, restrictive or obstructive.

[0203] As noted above, the quality of the classification may also be improved by appropriate pre-processing of the signal and thus at step S1508, there is an optional step of selecting the pre-processing to be applied to the received signal. The pre-processing may include filtering and temporal smoothing, for example to generate a “processed signal”. Alternatively, the pre-processing may additional comprise generating a gradient to provide a “gradient signal”. The received signal with its appropriate pre-processing may then be further processed to extract at least one feature to improve the classification by the binary classifiers at step S1508. For example, the signal may be processed to generate at least one shapelet as described above. The extracted feature is then processed through the first binary classifier to generate a first classification at step S1512.

[0204] There is an optional determination at step S1514 to determine whether the processing by the second binary classifier is required. It will be appreciated that if the normal vs abnormal classifier is used in the first step and the patient is classified as normal, there is no need to perform the second classification of obstructive v restrictive. Conversely, if the restrictive vs no-restrictive classifier is used in the first processing step, the second classifier is required to determine whether a patient who is classified as no-restrictive is normal or obstructive. If there is no further processing required, the output, e.g. the classification of normal may be output at step S1520. Otherwise, the at least one extracted feature is processed at step S1516 to generate the second classification.

[0205] The next step S1518 is to derive a single prediction per patient using the previous results are presented. One first approach, referred to as most common, involves selecting the class through majority voting and, in case of ties, choosing the class associated with the highest decision function value. The second approach, most certain, involves selecting the class with the highest decision function value for both the filter and reduced models. Such approaches may be needed when there is only one signal but are more likely to be needed when multiple signals are extracted from the same individual. Once a prediction (e.g. classification) is derived, it may be output at step S1520. The output classification may then be used to suggest a treatment for a patient as needed.

[0206] The next table summarises the most interesting results by highlighting the most promising models obtained for each voting scheme.Voting Scheme Most Certain Most CommonManoeuvre TB / TB FS / FSCombinationFilter model Restrictive-No Restrictive (SVC over Normal-Abnormal (SVC over gradient gradient signal of TB manoeuvre) signal of FS manoeuvre)Reduced Normal-Obstructive (SVC over Obstructive-Restrictive (Extra-Tree over model processed signal of TB manoeuvre) gradient signal of FS manoeuvre) Performance Balanced Accuracy: 0.786 (20 / 27 Balanced Accuracy: 0.832 (26 / 31samples predicted correctly). Per class: samples predicted correctly). Per class: 11 / 15, 5 / 8 and 4 / 4 normal, obstructive 16 / 17, 5 / 9 and 5 / 5 normal, obstructive and restrictive signals predicted and restrictive signals predictedcorrectly. correctly.

[0207] The outcomes presented in the table above demonstrate that combining multiple recordings for an individual effectively addresses the multiclass problem and enhances the performance compared to using a single recording. Specifically, employing the Most Common voting schemeincreased the balanced accuracy to a competitive 0.832 for the multiclass respiratory problem. This finding underscores the efficacy and relevance of the proposed methodology, even in the face of a highly imbalanced scenario and a limited dataset.

[0208] The above process uses an array of machine learning models to process signals derived from video recordings of individuals breathing normally or mimicking forced spirometry manoeuvres to classify them according to different respiratory conditions. Specifically, over 800 signals retrieved with clinical information on 175 individuals who participated in clinical studies conducted in both Japan and the UK were combined. A variety of modelling approaches, including support vector classifiers, K-nearest neighbours, boosting, and shapelets-based models were considered. These models are designed to distinguish between different conditions, such as patients suffering from a respiratory condition against normal patients, or patients suffering from restrictive vs. obstructive respiratory conditions, as well as to classify scenarios into multiple categories (normal, obstructive, restrictive). The top-performing models are hierarchical ensembles combining suitably designed shapelets-based binary classification problems and the use of signals derived from both manoeuvres (i.e., TB and FS) performed by participants to the considered clinical studies.

[0209] Despite the limited amount of data used to build these models, they have shown promising performance on independent validation sets. For the multiclass problem (normal vs. restrictive vs. obstructive), balanced accuracies of 0.754 were achieved (when analysing each recording independently) or 0.832 (when incorporating a voting scheme when more than one sample is available). For binary problems, such as restrictive vs. non-restrictive conditions, balanced accuracies and AUROC of 0.90 were achieved.

[0210] It will be appreciated that incorporating more recordings / patients would allow further extensions of the techniques described above. For instance, instead of relying solely on the signals extracted via the app, demographic information, such as gender, age, and anthropometric measurements, could be incorporated as input of the models to identify abnormal signal patterns across different demographic groups. Further processing of the input signals could also be done to extract features from these signals. Despite the limited and imbalanced data, the findings show that the signals extracted consistently perform well in various binary and multiclass respiratory classification problems, which are crucial in the current healthcare system scenario. Regarding the restrictive condition prediction, the amount of false negative cases was kept to a minimum regardless if the problem was considered multiclass or binary. This is highly significant considering the simplicity of sample extraction via the app, which could democratise access to a reliable respiratory problem screening tool.

[0211] In summary, the analysis of respiratory signals has various important clinical applications from general health monitoring to respiratory disease diagnosis and prognosis. While torso auscultation and spirometry are classical tests to evaluate relevant respiratory measurements, an array of tools exists with different levels of sophistication that might require more or less effort from the patient side and a varying degree of intervention on the clinician’s side. Even though acquisition of good quality signals is challenging, with the advent of new technologies, non-invasive contactlessreal-time monitoring techniques are becoming of increasing interest due to the advantages they offer. In fact, by not requiring contact with the subject, they do not interfere with the subject’s normal behaviour and as such they are much more likely to offer avenues for long-term use and continuous monitoring in both clinical and non-clinical settings.

[0212] The app described above is a prototype of such contactless technology and the outputs of this tool can be leveraged by sophisticated machine learning approaches as described above in order to provide promising classification results. These techniques could provide early detection indicators of respiratory health that could be made accessible to the wider population and could help identify individuals with undiagnosed respiratory conditions that should seek medical attention, especially when clinical access is not routinely sought.

[0213] At least some of the example embodiments described herein may be constructed, partially or wholly, using dedicated special-purpose hardware. Terms such as ‘component’, ‘processor’, ‘module’ or ‘unit’ used herein may include, but are not limited to, a hardware device, such as circuitry in the form of discrete or integrated components, a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks or provides the associated functionality. In some embodiments, the described elements may be configured to reside on a tangible, persistent, addressable storage medium and may be configured to execute on one or more processors. These functional elements may in some embodiments include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements.

[0214] Various combinations of optional features have been described herein, and it will be appreciated that described features may be combined in any suitable combination. In particular, the features of any one example embodiment may be combined with features of any other embodiment, as appropriate, except where such combinations are mutually exclusive. Throughout this specification, the term “comprising” or “comprises” means including the component(s) specified but not to the exclusion of the presence of others.

[0215] Attention is directed to all papers and documents which are filed concurrently with or previous to this specification in connection with this application and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference.

[0216] All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and / or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalentor similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

[0217] Although a few preferred embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes and modifications might be made without departing from the scope of the invention, as defined in the appended claims. The invention is not restricted to the details of the foregoing embodiment(s). The invention extends to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed.

Claims

CLAIMS1. A computer-implemented method for tracking a user’s chest movement, the method comprising:receiving video data of the user’s chest showing the user breathing, the video data comprising a plurality of two-dimensional images showing the user’s chest;locating at least one region of interest in a first image of the plurality of images, the at least one region of interest comprising a plurality of pixels;for each other image in the plurality of images,locating at least one tracking area in the image, wherein the located at least one tracking area corresponds to the at least one region of interest; andfor the at least one tracking area, determining a difference in location between the at least one tracking area in the image and the corresponding at least one tracking area in a previous image from the plurality of images, wherein determining the difference in location includes determining the difference in location along a direction perpendicular to a plane defined by the two-dimensional image;determining, using the determined location differences, movement of the user’s chest; and outputting a signal comprising information on the determined movement of the user’s chest.

2. The method as claimed in claim 1, further comprising:receiving depth information for each of the plurality of images; anddetermining the difference in location using the received depth information.

3. The method as claimed in claim 2, wherein locating the at least one region of interest and the at least one tracking area comprises locating at least one marker which has been positioned on a user’s chest; andwherein receiving depth information comprises calculating depth information using the located at least one marker.

4. The method as claimed in claim 3, wherein calculating depth information comprises:calculating a size of the at least one marker in the plurality of images;comparing the calculated size with the known size of the at least one marker; and estimating the depth information based on a difference between the calculated and known sizes.

5. The method as claimed in claim 4, wherein calculating depth information further comprises:determining a difference in location of at least one edge and / or at least one corner of the at least one marker between images andestimating the depth information using the determined difference in edge and corner locations.

6. The method as claimed in claim 3, wherein calculating depth information comprises: obtaining a normal vector oriented perpendicular to a surface of the marker for each marker in each of the plurality of images;determining a difference in normal vector orientation and location between images; and estimating the depth information using the determined difference in normal vector orientations and locations.

7. The method as claimed in claim 6, wherein determining the difference in location between images comprises:for the first image, setting the location of the normal vector as a reference location; and for each other image in the plurality of images, determining the difference in location relative to the reference location.

8. The method as claimed in any of claims 3 to 7 wherein locating at least one marker which has been positioned on a user’s chest comprises locating at least one marker which has been positioned on a user’s chest in a skin conformal way.

9. The method as claimed in claim 8 wherein locating at least one marker which has been positioned on a user’s chest in a skin conformal way comprises locating at least one marker which has been applied to a user’s hand which is resting on the user’s chest.

10. The method as claimed in any one of claims 3 to 9, further comprising locating a plurality of markers, wherein each marker encodes a unique identification pattern.

11. The method as claimed in claim 10, wherein locating a plurality of markers comprises locating at least one marker in a lower chest area on the user’s chest, and locating at least one marker in an upper chest area on the user’s chest.

12. The method as claimed in any claims 1 to 5, wherein locating at least one tracking area in each of the other images comprises, for each other image in the plurality of images,placing a search area at each location in the other image corresponding to a location of at least one region of interest and for each search area:defining multiple subareas within the search area, wherein each subarea has the same size and shape as the corresponding region of interest;calculating, for each of the defined subareas, a difference score between pixel values of the region of interest and pixel values of each defined subarea; andselecting, as the corresponding tracking area, the subarea which most closely matches the region of interest using the calculated difference score.

13. The method as claimed in any one of the preceding claims, comprising before determining a difference in location between the at least one tracking area in the image and a previous image, determining a confidence level in the located at least one tracking area; andonly when the confidence level is above a confidence threshold, continuing with the determining a difference step.

14. The method as claimed in any one of the preceding claim, wherein determining a difference in location between the corresponding tracking area in the image and a previous image comprises calculating a difference in position between at least one pixel in each tracking area.

15. The method as claimed in claim 2, wherein receiving depth information comprises receiving depth information from a depth capture device.

16. The method as claimed in claim 1 wherein:locating at least one region of interest in a first image of the plurality of images comprises locating at least one object overlaying the user’s chest; andfor each other image in the plurality of images, locating at least one tracking area in the image comprises locating at least one object overlaying the user’s chest corresponding to the at least one object in the first image.

17. The method as claimed in claim 16 wherein locating at least one object comprises locating one or more of a user’s hands.

18. The method as claimed in any one of the preceding claims wherein outputting a signal comprises outputting at least one of: a breathing rate, a loop diagram of the user’s breathing, and / or at least one metric of lung function.

19. The method as claimed in claim 18 wherein outputting at least one metric of lung function comprises:obtaining properties of at least one loop diagram of the user’s breathing, including at least an area under the curve for an inhalation and an area under the curve for an exhalation;calculating a ratio of the area under the curve for an exhalation and the area under the curve for an inhalation; andoutputting the calculated ratio.

20. A computer-implemented method for stratifying a user into a prognostic group for treatment of a respiratory disease, the method comprising:receiving at least one signal which is indicative of movement of a user’s chest;selecting a first binary classifier which outputs a first classification and a second classification; selecting a second binary classifier which outputs a third classification and a fourth classification, wherein at least one of the third and fourth classifications is different from the first and second classifications;extracting at least one feature from the received signal;inputting the at least one extracted feature into at least one of the first and second binary classifiers to obtain a classification of the received signal; andoutputting the obtained classification.

21. The method of claim 20, wherein the received at least one signal has been generated using the method of any one of claims 1 to 19.

22. The method of claim 20 or claim 21, wherein processing the received signal to extract at least one feature comprises processing the received signal to extract at least one shapelet.

23. The method of any one of claims 20 to 22, wherein inputting the at least one extracted feature into at least one of the first and second binary classifiers comprises inputting the at least one extracted feature into both of the first and second binary classifiers.

24. The method of any one of claims 20 to 23, further comprising:receiving a plurality of signal which are indicative of movement of a user’s chest; obtaining a classification for each of the plurality of signals; andderiving, from the obtained classifications, a single classification for the user.

25. The method of claim 24, wherein deriving a single classification comprises:selecting a most common obtained classification as the single classification orselecting a classification with a highest value as the single classification.

26. The method of any one of claims 20 to 25, further comprising:pre-processing the received at least one signal before extracting the at least one feature whereby the at least one feature is extracted from the pre-processed signal.

27. The method of claim 26, wherein pre-processing the received at least one signal comprises at least filtering and smoothing of the received at least one signal.

28. The method of claim 27, wherein pre-processing the received at least one signal further comprises generating a gradient signal.

29. The method of any one of claims 20 to 28, whereinselecting a first binary classifier comprises selecting a first binary classifier which outputs a first classification of normal and a second classification of abnormal; andselecting a second binary classifier comprises selecting a second binary classifier which outputs a third classification of obstructive and a fourth classification of restrictive,whereby the user is stratified into one of the following prognostic groups: normal, restrictive and obstructive.

30. The method of any one of claims 20 to 28, whereinselecting a first binary classifier comprises selecting a first binary classifier which outputs a first classification of restrictive and a second classification of not restrictive; andselecting a second binary classifier comprises selecting a second binary classifier which outputs a third classification of obstructive and a fourth classification of normal,whereby the user is stratified into one of the following prognostic groups: normal, restrictive and obstructive.

31. An apparatus for tracking a user’s chest movement, the apparatus comprising:an image capture device for capturing video data of the user’s chest that shows the user breathing; andat least one processor which is configured to track a user’s chest movement using the captured video data as set out in any one of claims 1 to 19.

32. The apparatus of claim 31, further comprising a depth capture device for providing depth information to the at least one processor.

33. The apparatus of claim 31 or claim 32, wherein the at least one processor is further configured to stratify a user into a prognostic group for treatment of a respiratory disease using the method of any one of claims 20 to 30.

34. A non-transitory carrier carrying a computer program which when implemented on a computer, causes the computer to implement the method of any one of claims 1 to 30.

Images