Device and method for determining joint angles of a subject

Abstract

A computer-implemented device for determining joint angles of a subject, which device comprises: a digital camera adapted to output digital video data comprising frames of image data in two-dimensions (2D) in an image plane of the digital camera; at least one machine learning model; and a memory for storing said digital video data in which the frames of image data contain at least one joint of the subject; the computer-implemented device adapted, for each frame of image data, to: determine using said at least one machine learning model a plurality of body landmarks of the subject, each body landmark comprising 2- dimensional (2D) co-ordinate data representing the co-ordinates of the body landmark in a first dimension and a second dimension within the image plane of the digital camera; perform a time-dependent analysis of said plurality of body landmarks to estimate a third-dimension co-ordinate for each body landmark in said frames of image data, said third dimension mutually perpendicular to said first and second dimensions; use said third dimension co-ordinates to convert said 2D co-ordinate data into three-dimensional (3D) co-ordinate data for each body landmark; use said 3D co-ordinate data of said plurality of said body landmarks to determine a 3D joint angle of said at least one joint of the subject within respective frames of image data; and output data representing said 3D joint angle.

Description

[0001] P5642PC00-MH8NG

[0002] DEVICE AND METHOD FOR DETERMINING JOINT ANGLES OF A SUBJECT

[0003] TECHNICAL FIELD

[0004] Some embodiments of the invention relate to a computer-implemented devices and methods for determining joint angles of a subject.

[0005] BACKGROUND

[0006] The accurate analysis of human motion and posture is crucial in clinical diagnostics, sports training, rehabilitation, physiotherapy, and workplace ergonomics. Presently available systems use various tools to measure joint angles of a subject, from which other metrics may be evaluated such as balance, gait analysis, or static posture assessment. Such separate tools often require manual calibration, specialised hardware, and provide only limited feedback in real-time.

[0007] Some attempts have been made to analyse human motion and posture with specialised rigs of 3D cameras that surround the user. The user must wear a special suit upon which body landmarks (e.g. elbow joint, shoulder joint, etc) are provided. These rigs are very expensive to build and operate, do not provide real-time feedback, and are not suitable for use at home or clinical environment.

[0008] SUMMARY OF THE INVENTION

[0009] According to some embodiments there is provided a computer-implemented device for determining joint angles of a subject. The device may comprise a digital camera adapted to output digital video data comprising frames of image data in two-dimensions (2D) in an image plane of the digital camera. The device may comprise at least one machine learning model. The device may comprise a memory for storing said digital video data in which the frames of image data contain at least one joint of the subject. The computer-implemented device may be adapted, for each frame of image data, to determine using said at least one machine learning model a plurality of body landmarks of the subject. Each body landmark may comprise 2-dimensional (2D) co-ordinate data representing the co-ordinates of the body landmark in a first dimension and a second dimension within the image plane of the digital camera. The device may be adapted to perform a depth inference algorithm, such as a monocular depth inference algorithm, or a time-dependent analysis of said plurality of body landmarks to estimate a third-dimension co-ordinate for each body landmark in said frames of image data, said third dimension mutually perpendicular to said first and second dimensions. The device may be adapted to use said third dimension co-ordinates to convert said 2D co-ordinate data into three-dimensional (3D) co-ordinate data for each body landmark. The device may be adapted to use said 3D co-ordinate data of said plurality of said body landmarks to determine a 3D joint angle of said at least one joint of the subject within respective frames of image data. The device may be adapted to output data representing said 3D joint angle.P5642PC00-MH8NG

[0010] In some embodiments the subject may be a human or an animal having a skeleton, such as a domesticated animal. In some embodiments the computer-implemented device and methods described herein may be useful in fields including analysis of human motion and posture, clinical diagnostics, sports training, rehabilitation, physiotherapy, and workplace ergonomics.

[0011] In some embodiments the computer-implemented may be further adapted to identify and discard from said plurality of body landmarks any unreliable body landmark in said plurality of body landmarks.

[0012] In some embodiments the unreliable body landmark may comprise a body landmark that would result in determining a joint angle outside an expected range and / or a body landmark that is missing or occluded in said frame of image data.

[0013] In some embodiments the said machine learning model may be adapted to output a confidence value with each body landmark detected. The confidence value may represent how likely it is that the body landmark has been correctly identified. To identify and discard an unreliable body landmark said computer-implemented device may be further adapted to determine whether said confidence value greater or less than a threshold, and to discard said body landmark as unreliable if said confidence value is lower than said threshold. The confidence value may be a value between 0 and 1. The confidence threshold may be between approximately 0.4 and 0.6. In some embodiments the confidence threshold may be approximately 0.5.

[0014] In some embodiments to identify and discard an unreliable body landmark said computer-implemented device may be further adapted to use said 2D co-ordinate data for a plurality of said body landmarks to estimate a 2D joint angle of said at least one joint of the subject within respective frames of image data, each said 2D joint angle lying in said image plane of the digital camera.

[0015] In some embodiments to identify and discard an unreliable body landmark said computer-implemented device may be further adapted to determine whether said 2D joint angle is an outlier compared to previous calculations of said 2D joint angle in said digital video data.

[0016] In some embodiments to identify and discard an unreliable body landmark said computer-implemented device may be further adapted to perform a statistical analysis of said 2D joint angle, such as determining whether said joint angle is within k standard deviations of 2D joint angles determined previously in said digital video data.

[0017] In some embodiments to perform said time-dependent analysis said computer-implemented device may be further adapted to use a Kalman filter, exponential smoothing or an AutoP5642PC00-MH8NG

[0018] Regressive Integrated Moving Average (ARIMA) to estimate said third-dimension coordinate for each frame of image data.

[0019] In some embodiments the computer-implemented device may be further adapted to determine a first expected value of said third-dimension co-ordinate using anthropometric data and to input said first expected value into said time-dependent analysis.

[0020] In some embodiments said first expected value may relate to a body landmark on the torso of the subject, the computer-implemented device may be adapted to adjust said first expected value of said body landmark on the torso by a scaling factor, and to use said adjusted first expected value to determine a first expected value of said third dimension for another body landmark, such as a body landmark on a limb of the subject.

[0021] In some embodiments the machine learning model may be adapted to output a third dimension estimate of each body landmark. The computer-implemented device may be further adapted to discard each said third-dimension estimate and to use only said 2-dimensional (2D) co-ordinate data output by said machine learning model. Some machine learning models adapted for pose estimation output a depth dimension estimate of each body landmark that is given relative to other body landmarks, such as the hips. Such an output depth estimate may be discarded.

[0022] In some embodiments the computer-implemented device may comprise a processor. The device may be further adapted to use a separate thread of execution within said processor to determine said plurality of body landmarks with said machine learning model.

[0023] In some embodiments the computer-implemented device may be further adapted to display on a graphical user interface any body landmarks representing a joint angle identified in said frame of image data and / or any indication based on said output 3D joint angle, before processing a next frame of image data. The indication may be an indication to the user on the graphical user interface whether the determined 3D joint angle is within a normal range of values. The indication may be displayed substantially in real-time enabling the user to move the joint in question to see on immediately on the graphical user interface when the joint is in a position within the normal range of values. This may be useful to help the user adjust their posture in real-time to an improved posture, for example.

[0024] In some embodiments the computer-implemented device may be adapted to store each output 3D joint angle, optionally with other data such a timestamp of the output 3D joint angle. In some embodiments, each output 3D joint angle can be appended to a data structure such as a CSV data structure.

[0025] In some embodiments the computer-implemented device may be further adapted to read a body landmark selection mask representing at least one joint angle of the subject. TheP5642PC00-MH8NG

[0026] device may be adapted to discard body landmarks output by the machine learning model that are not identified in said body landmark selection mask.

[0027] In some embodiments the body landmark selection mask may identify body landmarks corresponding to one or more joint angle useful for analysing posture and / or movement of the subject.

[0028] In some embodiments the body landmarks may represent upper and lower body joints of the subject. The computer-implemented device may be further adapted to determine an average value of said 3D joint angles over a period of time. The computer-implemented device may be adapted to store said average values of said 3D joint angles alongside values of said 3D joint angle determined for each frame of image data.

[0029] In some embodiments the body landmarks represent a posture parameter of the subject, the computer-implemented device further adapted to overlay on said digital video data an indication when said posture parameter is within a normal or good range.

[0030] In some embodiments the computer-implemented device may be further adapted to overlay said indication substantially in real-time. In some embodiments, the indication may be shown to the subject almost instantaneously, i.e. substantially in real-time, on a 'live' video feed so that, as the subject changes posture the indication changes to help the subject adopt a good posture.

[0031] In some embodiments said body landmarks may comprise the nose, left and right shoulder and left and right hips, and optionally left and right eyes. The computer-implemented device may be further adapted to use said body landmarks to determine neck angle and head tilt, and to compare said neck angle and head tilt to said normal range.

[0032] In some embodiments said body landmarks represent a lower body and a core of the subject. The computer-implemented device may be further adapted to use said body landmarks to determine a centre of mass of the subject with each frame of image data.

[0033] In some embodiments the computer-implemented device may be further adapted to determine said centre of mass using a weighted average of body segments corresponding to said body landmarks.

[0034] In some embodiments the computer-implemented device may be further adapted to perform a time-series analysis or a depth inference algorithm, such as a monocular depth inference algorithm, of said centre of mass.

[0035] In some embodiments the computer-implemented device may be further adapted to determine a stability index for the subject by determining a horizontal distance between aP5642PC00-MH8NG

[0036] midpoint of a line joining body landmarks of the feet of the subject and the centre of mass projected onto the ground.

[0037] In some embodiments the computer-implemented device may be further adapted to track said stability index substantially continuously.

[0038] In some embodiment the computer-implemented device may be further adapted to determine an extrapolated centre of mass for the subject indicating where the centre of mass is moving. The computer-implemented device may be further adapted to determine the extrapolated centre of mass using co-ordinates of the centre of mass adjusted by a term proportional to a measured velocity of the centre of mass (derived for example by determining how the centre of mass co-ordinates change between frames of image data).

[0039] In some embodiments said body landmarks may represent a lower body of the subject including hip and ankle. The computer-implemented device may be adapted to use said body landmarks to determine leg swing direction by comparing x co-ordinates of the hip and ankle.

[0040] In some embodiments said body landmarks may represent a core and a lower body of the subject. The computer-implemented device may be further adapted to detect sit-to-stand transitions by determining changes between frames of image data in the 3D joint angles of the core and lower body.

[0041] In some embodiments said body landmarks may represent a lower body of the subject. The computer-implemented device may be further adapted to determine gait metrics of the subject during a walking motion, such as step length, stride length, cadence and walking speed.

[0042] In some embodiments said body landmarks may represent substantially a full body of the subject. The computer-implemented device may be further adapted to determine a head to ankle ratio, which may be dependent on a distance between body landmarks of the head and ankle and an estimated body height of the subject.

[0043] In some embodiments said memory may store a range of normal values for each said 3D joint angle. The computer-implemented device may be further adapted to determine whether said 3D joint angle lies within said range. As used herein 'normal' values may be those values associated with a patient or group of patients such as a population or part of population.

[0044] In some embodiments said 3D joint may angle comprises an angle between two segments formed by three body landmarks. Optionally said angle may comprise an internal angle.P5642PC00-MH8NG

[0045] In some embodiments the computer-implemented device may be in the form of a personal computer such as a smartphone, tablet, gaming device, laptop or desktop computer.

[0046] In some embodiments said 2D digital camera may integrated into a body of the computer-implemented device.

[0047] In some embodiments the computer-implemented device may be further adapted to perform pre-processing on said digital video data before frames of image data are input into said machine learning model. The pre-processing may include at least one of: image rotation to normalise an orientation of the subject; resizing frames of image data to the same size; maintaining aspect ratio; normalising pixel values; converting a colour space of the frames of image data to the same format, such as RGB; white balancing; histogram equalisation; and denoising.

[0048] According to some aspects of the invention there is provided a computer-implemented method for determining joint angles of a subject. The method may comprise the step of receiving from a digital camera digital video data comprising frames of image data in two-dimensions (2D) in an image plane of the digital camera and containing at least one joint of the subject. The method may comprise storing said digital video data. The method may comprise steps for each frame of image data. Those steps may include determining using at least one machine learning model a plurality of body landmarks of the subject. Each body landmark may comprise 2-dimensional (2D) co-ordinate data representing the co-ordinates of the body landmark in a first dimension and a second dimension within the image plane of the digital camera. Those steps may include performing a time-dependent analysis of said plurality of body landmarks to estimate a third-dimension co-ordinate for each body landmark in said frames of image data. The third dimension may be mutually perpendicular to said first and second dimensions. Those steps may include using said third dimension coordinates to convert said 2D co-ordinate data into three-dimensional (3D) co-ordinate data for each body landmark. Those steps may include using said 3D co-ordinate data of said plurality of said body landmarks to determine a 3D joint angle of said at least one joint of the subject within respective frames of image data. Those steps may include outputting data representing said 3D joint angle.

[0049] In some embodiments there is provided a data processing apparatus comprising a memory and a processor for carrying the method of claim 43. The data processing apparatus may be adapted to comprise any of the features of the computer-implemented device set out above, or as described or as claimed anywhere herein.

[0050] In some embodiments the data processing apparatus may comprise a virtual server, bare-metal server or cloud computing platform.P5642PC00-MH8NG

[0051] According to some aspects of the invention there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the computer-implemented method as set out above.

[0052] According to some aspects of the invention there is provided a computer program product comprising a computer program as set out above.

[0053] In some embodiments the computer program product may be in the form of an application downloadable from a server. For example the computer program product may be an app downloadable from an app store.

[0054] According to some aspects of the invention there is provided a computer-readable data carrier having stored thereon a computer program product as set out above.

[0055] According to some aspects of the invention there is provided a computing device, such as a hand-held computing device, comprising the computer-readable data carrier as set out above.

[0056] According to some aspects of the invention there is provided a data carrier signal carrying the computer program product set out above.

[0057] Throughout the description and claims of this specification, the words "comprise" and "contain" and variations of the words, for example "comprising" and "comprises", mean "including but not limited to", and do not exclude other components, integers or steps. Moreover the singular encompasses the plural unless the context otherwise requires: in particular, where the indefinite article is used, the specification is to be understood as contemplating plurality as well as singularity, unless the context requires otherwise.

[0058] Preferred features of each aspect of the invention may be as described in connection with any of the other aspects. Within the scope of this application it is expressly intended that the various aspects, embodiments, examples and alternatives set out in the preceding paragraphs, in the claims and / or in the following description and drawings, and in particular the individual features thereof, may be taken independently or in any combination. That is, all embodiments and / or features of any embodiment can be combined in any way and / or combination, unless such features are incompatible.

[0059] BRIEF DESCRIPTION OF THE DRAWINGS

[0060] One or more embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0061] Figure 1 is a schematic diagram of showing an embodiment of a computer-implemented device according to the present invention during use;P5642PC00-MH8NG

[0062] Figure 2 is a schematic diagram of an embodiment of computer-implemented device according to the present invention;

[0063] Figure 3 is a schematic diagram showing numbered body landmarks on the human body;

[0064] Figure 4 is a flowchart illustrating steps in a computer-implemented method according to an embodiment of the present invention;

[0065] Figure 5 is a continuation of the flowchart of Figure 4;

[0066] Figures 5A - 5B are examples visualisations and textual suggestions based on the output of the computer-implemented method of Figures 4 and 5;

[0067] Figure 6 is a schematic diagram illustrating selection of various body landmark selection masks;

[0068] Figures 6A to 6G are diagrams illustrating the body landmarks selected by body landmark selection masks of Figure 6 used to perform certain joint angle calculations, and then additional steps of using those joint angle calculations to determine various biomechanical parameters of the subject;

[0069] Figure 6H show various plots of 3D joint angles against time; and

[0070] Figure 7 is a schematic diagram of a computer-implemented device according to an embodiment of the present invention in use with an optional cloud computing platform and portal.

[0071] DETAILED DESCRIPTION

[0072] The following detailed description and disclosure illustrates by way of example and not by way of limitation. This description will clearly enable one skilled in the art to make and use the disclosed systems and methods, and describes several embodiments, adaptations, variations, alternatives, and uses of the disclosed systems and methods. As various changes could be made in the above constructions without departing from the scope of the disclosures, it is intended that all matter contained in the description or shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

[0073] Throughout this disclosure the term "computer" means hardware which generally implements functionality provided by digital computing technology, particularly computing functionality associated with microprocessors. The term "computer" is not intended to be limited to any specific type of computing device, but it is intended (unless otherwise qualified) to be inclusive of all computational devices including, but not limited to: processing devices, microprocessors, personal computers, desktop computers, laptopP5642PC00-MH8NG

[0074] computers, workstations, terminals, servers, clients, portable computers, handheld computers, cell phones, mobile phones, smart phones, tablet computers, server farms, hardware appliances, minicomputers, mainframe computers, video game consoles, handheld video game products, and wearable computing devices including, but not limited to eyewear, wristwear, pendants, fabrics, and clip-on devices.

[0075] As used herein, a "computer" is necessarily an abstraction of the functionality provided by a single computer device outfitted with the hardware and accessories typical of computers in a particular role. By way of example and not limitation, the term "computer" in reference to a laptop computer would be understood by one of ordinary skill in the art to include the functionality provided by pointer-based input devices, such as a mouse or track pad, whereas the term "computer" used in reference to an enterprise-class server would be understood by one of ordinary skill in the art to include the functionality provided by redundant systems, such as RAID drives and dual power supplies.

[0076] It is also well known to those of ordinary skill in the art that the functionality of a single computer may be distributed across a number of individual machines. This distribution may be functional, as where specific machines perform specific tasks; or, balanced, as where each machine is capable of performing most or all functions of any other machine and is assigned tasks based on its available resources at a point in time. Thus, the term "computer" as used herein, can refer to a single, standalone, self-contained device or to a plurality of machines working together or independently, including without limitation: a network server farm, "cloud" computing system, software-as-a-service (SAAS), or other distributed or collaborative computer networks. Thus, to the extent this disclosure describes systems or methods as being performed by or on a computer, a person of ordinary skill in the art will understand that, unless specified otherwise, the systems and methods may be implemented on a single device or distributed across multiple devices.

[0077] Those of ordinary skill in the art also appreciate that some devices not conventionally thought of as "computers" nevertheless exhibit the characteristics of a "computer" in certain contexts. Where such a device is performing the functions of a "computer" as described herein, the term "computer" includes such devices to that extent. Devices of this type include, but are not limited to: network hardware, print servers, file servers, NAS and SAN, load balancers, loT devices, smart devices, and other hardware capable of interacting with the systems and methods described herein in the matter of a conventional "computer."

[0078] Throughout this disclosure, the term "software" refers to code objects, program logic, command structures, data structures and definitions, source code, executable and / or binary files, machine code, object code, compiled libraries, implementations, algorithms, libraries, or any instruction or set of instructions capable of being executed by a computer processor, or capable of being converted into a form capable of being executed by a computerP5642PC00-MH8NG

[0079] processor, including, without limitation, virtual processors, or by the use of run-time environments, virtual machines, and / or interpreters.

[0080] Those of ordinary skill in the art recognize that although software is traditionally stored in a non-transitory computer-readable medium and loaded into memory on demand for execution, software can also be wired or embedded into hardware, including, without limitation, onto a microchip, and still be considered "software" within the meaning of this disclosure. For purposes of this disclosure, software includes, without limitation: instructions stored or storable in hard drives, RAM, ROM, flash memory BIOS, CMOS, mother and daughter board circuitry, hardware controllers, USB controllers or hosts, peripheral devices and controllers, video cards, audio controllers, network cards, Bluetooth® and other wireless communication devices, virtual memory, storage devices and associated controllers, firmware, and device drivers. The systems and methods described here are contemplated to use computers and computer software typically stored in a computer- or machine-readable storage medium or memory.

[0081] Throughout this disclosure, the term "network" generally refers to a voice, data, or other telecommunications network over which computers communicate with each other. The term "server" generally refers to a computer providing a service over a network, and a "client" generally refers to a computer accessing or using a service provided by a server over a network. Those having ordinary skill in the art will appreciate that the terms "server" and "client" may refer to hardware, software, and / or a combination of hardware and software, depending on context. Those having ordinary skill in the art will further appreciate that the terms "server" and "client" may refer to endpoints of a network communication or network connection, including, but not necessarily limited to, a network socket connection. Those having ordinary skill in the art will further appreciate that a "server" may comprise a plurality of software and / or hardware servers delivering a service or set of services as described elsewhere herein. Those having ordinary skill in the art will further appreciate that the term "host" may, in noun form, refer to an endpoint of a network communication or network (e.g., "a remote host"), or may, in verb form, refer to a server providing a service over a network ("host a website"), or an access point for a service over a network.

[0082] Some embodiments of the invention relate to a computer-implemented method and device for determining a joint angle of a subject. The joint of a subject, also known as an articulation, is the connection between bones, ossicles, or other hard structures in the body which link the subject's skeletal system into a functional whole. Joints generally allow for different degrees and types of movement. Joints play an important role in the human body, contributing to movement, stability and overall function. They are essential for mobility and flexibility, connecting bones and facilitating a wide range of motions, from simple bending and stretching to complex actions like running and jumping. Beyond enabling movement,P5642PC00-MH8NG

[0083] joints provide structural support and stability to the skeleton, helping to maintain posture, balance, and the ability to bear weight during daily activities.

[0084] The joint angle may be the full angle through which a subject can articulate the joint comfortably. The joint angle may also be an angle through which the subject can articulate the joint when performing an activity such as walking, running, playing sport, etc. The joint angle may also be an angle through which the subject can articulate the joint when performing a specific movement, exercise or test, often under the instruction of a healthcare professional. Accordingly, the joint angle may not be the full angle through which the joint can be articulated in isolation. The joint angle may be an angle defined between parts of the body when the subject is relatively still, such as sitting or standing.

[0085] Measurement of one or more joint angle when the subject is relatively still can be useful for assessment of posture, for example. Accordingly, and as used herein, joint angle may refer to a dynamic angle defined by articulation of a joint and also to a static angle defined by a joint when the subject is relatively still. As used herein 'static' may mean that the joint angle does not change significantly with time compared to the rate of change of joint angle when the subject is moving. Examples of static joint angles are angles formed by joints when a subject is sitting at a desk or standing still. Examples of dynamic joint angles are when the subject is moving.

[0086] In general, measuring a joint angle of a subject can provide useful information about the functioning of the joint and the surrounding muscles, tendons and other structures, as well facilitating assessment of posture. Measuring a joint angle can provide useful data to help the subject and / or clinician diagnose and manage a range of musculoskeletal issues, and to help with rehabilitation, sports training, etc.

[0087] In some embodiments, software may be provided for measuring joint angles of subject using conventional hardware, such as a smartphone having an integrated 2D digital camera. The software may comprise a pre-trained machine learning model suitable for identifying various body landmarks of a subject in image frames of digital video data from the 2D digital camera, such as pre-recorded video or a 'live feed' from the 2D digital camera. A body landmark is the location of one or more joint of the subject. The machine learning model may also estimate the 3D co-ordinates of each body landmark, including a depth or z co-ordinate relative to subject. The software may remove the co-ordinate representing relative depth of each body landmark and then perform a time-series analysis (e.g. a Kalman filter) or a depth inference algorithm, such as a monocular depth inference algorithm, to estimate a new co-ordinate representing depth of each body landmark from an image plane of the 2D digital camera. An initial estimate for the depth of the body landmark for input into the time-series analysis or a depth inference algorithm may be based on known anthropometric data. The new co-ordinate representing depth of each body landmarkP5642PC00-MH8NG

[0088] from the image plane of the 2D camera is combined with the two co-ordinates from the machine learning model to form 3D co-ordinates for each body landmark. The 3D coordinates of a set of body landmarks are then processed by the software to determine one or more joint angle of the subject. For example, three body landmarks (e.g. representing shoulder, elbow and wrist) may be used to define two vectors corresponding to segments of the body (such as arm and forearm). The software may determine the angle between the two vectors and this is the joint angle in real-world 3 dimensions (as opposed to the projection of this angle onto the image plane of the 2D camera). The software may repeat this process for subsequent image frames in the digital video data and store the calculated joint angles for subsequent use and / or display an indication about the calculated joint angles on a graphical user interface, such as whether the joint angle is within a normal range. The software may perform the aforementioned steps substantially in real-time.

[0089] Before the time-series analysis / depth inference algorithm, the software may also remove unreliable body landmarks that it detects. An advantage is that the estimate of the depth of the body landmark is improved as unreliable data is not input into the time-series analysis / depth inference algorithm, which may otherwise affect the estimate of the depth co-ordinate. An unreliable body landmark may be one that is occluded in the image frame, has moved outside the image frame, or the co-ordinates are not accurately determined by the machine-learning model leading to an unrealistic joint angle determination (such as an outlier).

[0090] Figure 1 illustrates an embodiment of a computer-implemented device 100 during use with a subject 102. In this embodiment the computer-implemented device 100 is a smartphone comprising at least one digital camera (not shown), which may be in-built into the body of the smartphone, either on the front and / or back of the smartphone. In some embodiments the digital camera used may be the user-facing digital camera on the same side as the GUI 104. In this way the user can watch their movements and receive posture and motion feedback in real time.

[0091] The computer-implemented device 100 may be any computer-implemented device comprising or having access to a digital camera including, but not limited to, a camera phone, a desktop or laptop computer, a tablet, a gaming device, and any wearable device such as a smartwatch and smart glasses. Generally, such computer-implemented devices may be hand-held or wearable by a person, but this is not essential. The computer-implemented device 100 may have access (e.g. wired or wirelessly) to a remote digital camera, such as a webcam. The computer implemented device 100 may take a different form-factor, such as a hand-held digital camera or a portable video recording device. In general, the computer-implemented device 100 may be any device capable of recording digital video data.P5642PC00-MH8NG

[0092] In Figure 1, the computer-implemented device 100 is being used to record the subject 102 a field of view 103 of its digital camera. Figure 1 shows two instants in time, to and ti, with to before ti. In this example the subject 102 has moved their arms between times to and ti. However, it is not essential for the subject 102 to move whilst being recorded with the digital camera. The subject 102 may be relatively still, for example sitting. The subject 102 may also be performing a particular pose or a particular action (e.g. a sports movement such as walking, running, a golf swing, a cricket shot, etc.). The subject 102 is usually a human, but it is conceivable that in some embodiments the subject may be any animal having a skeleton, and any domestic animal such as a dog, cat or horse.

[0093] The output from the digital camera can be seen in real-time on the graphical user interface (GUI) 104 of the smartphone, although this is not essential. A co-ordinate system 106 is fixed to the image plane of the digital camera. In this embodiment the image plane of the digital camera is substantially co-planar with a plane of GUI 104. This may not be the case in all embodiments and / or if the computer-implemented device 100 does not comprise a GUI 104 at all. As shown, the co-ordinate system 106 comprises axes x and y. The digital camera outputs digital video data comprising frames of image data in two dimensions (2D), corresponding to x and y, in the image plane of the digital camera. A co-ordinate axis z is shown in Figure 1 which represents depth, e.g. a perpendicular distance between the image plane of the camera and the subject 102. The digital video data does not contain any information about z, and the scene of the subject 102 is effectively a projection onto the image plane of the digital camera. Throughout this disclosure such a digital camera will be referred to as a 2D digital camera.

[0094] Figure 2 illustrates some of the components of computer-implemented device 100. The computer-implemented device 100 may comprise a case 108 housing a processor 110, a computer memory 112, the aforementioned GUI 104 and 2D digital camera 114. The computer-implemented device 100 may also comprise one or more wireless network interface (not shown), such as a cellular network, WLAN, Bluetooth, etc., for reception and transmission of data with external networks. The processor 110 interfaces with all the aforementioned components to process (store, access, etc.) digital data. For example, digital video data may be processed by the processor 110 and stored in the memory 112.

[0095] The memory 112 stores computer executable instructions that when executed by the CPU 44 bring into operation an operating system and various individual mobile application software (or 'apps') 111. To obtain mobile application software, the computer executable instructions may be downloaded from an app store, such as Google Play and the Apple App Store. The user may interact with the mobile application software 111 via the graphical user interface 104. Embodiments of the invention may be provided in the form of such mobileP5642PC00-MH8NG

[0096] application software, the functions of which will be described in greater detail below and which will be referred to as app 111.

[0097] The memory 112 also stores a machine learning ('ML') model 116. The ML model 116 may be part of the app 111 that is downloaded by a user from an app store. The ML model 116 may be any ML model or combination of ML models pre-trained for detecting within the digital video data body landmarks on the body of the subject 102. In general, a body landmark may be the approximate location of certain parts of the body. Figure 3 provides an example in which body landmarks 118 are indicated by numbered circles with the following key:

[0098] 0 - nose

[0099] 1 - left eye (inner)

[0100] 2 - left eye

[0101] 3 - left eye (outer)

[0102] 4 - right eye (inner)

[0103] 5 - right eye

[0104] 6 - right eye (outer)

[0105] 7 - left ear

[0106] 8 - right ear

[0107] 9 - mouth (left)

[0108] 10 - mouth (right)

[0109] 11 - left shoulder

[0110] 12 - right shoulder

[0111] 13 - left elbow

[0112] 14 - right elbow

[0113] 15 - left wrist

[0114] 16 - right wrist

[0115] 17 - left pinky

[0116] 18 - right pinky

[0117] 19 - left index

[0118] 20 - right index

[0119] 21 - left thumb

[0120] 22 - right thumb

[0121] 23 - left hip

[0122] 24 - right hip

[0123] 25 - left knee

[0124] 26 - right knee

[0125] 27 - left ankle

[0126] 28 - right ankleP5642PC00-MH8NG

[0127] 29 - left heel

[0128] 30 - right heel

[0129] 31 - left foot index

[0130] 32 - right foot index

[0131] Embodiments of the invention are not limited to the combination and number of body landmarks shown in Figure 3. A ML model may be trained to identify different numbers of body landmarks in different combinations.

[0132] During use, frames of image data captured by the 2D digital camera may be input to the ML model 116 stored on the computer-implemented device 100. For each frame of image data, the ML model 116 outputs data representing body landmarks that are visible. It may be that within a single frame none, some or all the body landmarks may be visible.

[0133] In some embodiments, the ML model 116 may be Google's MediaPipe Pose Landmarker, available from https; / / a Lgpogle, dev / edge / media ipe / solutjons / yjsj n / pose tend marker. An advantage of using a pre-trained ML model 116 is that it is ready to run by the app 111 on standard hardware such as a smartphone. Examples of other machine learning models which could be used include, but are not limited to, Openpose and poseNet.

[0134] Figure 3 illustrates steps in a computer-implemented method 200 for determining joint angles of the subject 102 performed by app 111 stored on the computer-implemented device 100. At step S201 digital video data (either 'live' from the 2D digital camera 114 or recorded) is received. The digital video data may be preprocessed before input to the ML model 116. The preprocessing may include some or all of image rotation, resizing, normalisation and colour space conversion (e.g. YCbCr to RGB). Some preprocessing steps may be helpful for ensuring that the input to the machine learning model 116 is both consistent and robust, particularly in a biomechanical evaluation where precision is important. When the 2D digital camera 114 is not stationary or if the subject's position changes, image rotation can help normalise the subject's orientation. This assists the machine learning model 116 to process image frames with body landmarks consistently aligned with the 2D digital camera's coordinate system. The machine learning model 116 may also require input image frames of a specific size (e.g. pixel width and height). Resizing the images standardises the input resolution, reducing variability in landmark detection that could arise from different image scales. Maintaining the aspect ratio may also be helpful to avoid distorting anatomical features. Normalising pixel values (for instance, scaling them to a 0-1 range) helps to reduce variations caused by differences in lighting and contrast. This is very helpful to ensure that the performance of the machine learning model 116 is substantially unaffected by external conditions. Many machine learning models, particularly those used for pose estimation, are trained on RGB images. If the 2D digital camera 114 outputs digital video data in a different colour space (such as YCbCr or BGR), converting toP5642PC00-MH8NG

[0135] RGB ensures that the model receives data in the same format as it was trained on. This can have an important influence detection accuracy. In challenging lighting conditions, such as recording outdoors on a sunny day or in snow where the light is excessively bright, further steps, such as white balancing or histogram equalisation, may be implemented to enhance the quality of the input image. Depending on the quality of the 2D digital camera 114, applying denoising filters can also improve landmark detection by minimising high-frequency noise. Since the method operates on digital video data, maintaining consistent preprocessing across all frames is helpful to ensure temporal consistency in the detected landmarks, thereby enhancing the performance of time-series analyses described further below.

[0136] In the following the method 200 is described for a single image frame of the digital video data. It is to be noted that the method 200 is repeated for each frame of image data. At step S202 each frame of image data is input to the ML model 116 to perform body landmark detection.

[0137] Google's MediaPipe Pose Landmarker is optimised for real-time performance and may be used in some embodiments. Other machine learning models trained for the task of pose estimation may be used in other embodiments. MediaPipe Pose Landmarker uses a two-stage pipeline (first detect the subject 102, then track body landmarks) and doesn't run the full detector on every image frame. The present inventors have discovered that, whilst this approach speeds up inference significantly compared to starting the detector each time, it is possible to cause a bottleneck in performance of the computer-implemented device 100 by processing frames on the main thread (freezing the GUI 104) or by doing a lot of computations on each frame reducing runtime performance.

[0138] To improve the throughput of image frames, the body landmark detection process is launched in a separate execution thread so that the GUI 104 remains responsive.

[0139] Furthermore, the ML model 116 is launched once and re-used for all image frames of the digital video data rather than re-creating it repeatedly to avoid extra overhead. This enables the application to be run on modest hardware (e.g. commercially available smartphones).

[0140] An output from the ML model 116 for each frame of image data comprises an identification of each body landmark (see Figure 3) with its 3D co-ordinates (x, y and z, see Fig. 1) found in the frame. Of note, a z value is estimated by the ML model for each body landmark 118 but is anchored to the subject's hip plane rather than an absolute depth between the image plane of the digital camera 114 and the body landmark 118. A problem recognized by the present inventors is that the z values output by the ML model can jitter or skew if the subject 102 moves toward or away from the 2D digital camera 114 and / or if some of body landmarks are occluded at any time.P5642PC00-MH8NG

[0141] At step S203 body landmark selection mask is read in by the app 111. Aspects of the body landmark selection mask will be described in greater detail below. However, in general, the body landmark selection mask represents a sub-set, or choice, of the 33 available body landmarks in which a user is interested to know the or each joint angle, for example to assist with assessment of posture or motion of the subject. For example, the user may be interested only in body landmarks numbered 12 (right shoulder), 14 (right elbow) and 16 (right wrist) to reveal movement of the elbow joint. The choice of body landmarks may include more than one joint angle. All other body landmarks detected by the ML model 116 at step S202 are discarded at step S204, leaving only those indicated by the body landmark selection mask. However, for the first image frame some key landmarks are retained. For the initial Kalman filter estimate, the most stable and consistently detected landmarks may be retained as key landmarks. The first estimate may be derived primarily from torso landmarks because they are less affected by dynamic motion of the subject. The following body landmarks have been found to be stable and consistently detected.

[0142] Nose (landmark 0)

[0143] Left Shoulder (landmark 11)

[0144] Right Shoulder (landmark 12)

[0145] Left Hip (landmark 23)

[0146] Right Hip (landmark 24)

[0147] These landmarks form a robust baseline for estimating the depth (z) of the torso. The rationale is that the torso tends to move less and remains relatively consistent across frames, providing a stable reference. Once the torso's z-value is established, the z-values of other landmarks (such as elbows, knees, and wrists) may be adjusted using small empirically determined correction factors (5).

[0148] If the hip landmarks are unavailable or unreliable due to occlusions or specific poses, it may be possible to fall back on just the nose and shoulder body landmarks. However, omitting the hips might reduce the overall accuracy of the depth estimation, as the hips provide a very useful reference point for the lower torso. In scenarios where additional stability is required, one might also consider incorporating eye body landmarks to better capture head orientation and further refine the estimate.

[0149] At step S205 the method 200 performs data filtering for quality. Firstly, the method 200 checks a confidence value for each selected body landmark. Rather than simply trusting that all the selected body landmarks are 'good' the method 200 checks a confidence value for each selected body landmark: if any of the selected body landmarks have a low confidence value (e.g. < 0.5 has been found suitable for most conditions), the body landmark is deemed unreliable and the method returns 'None' for that joint angle of that image frame. This avoids using unreliable body landmarks in joint angle calculations, actingP5642PC00-MH8NG

[0150] as an error-correction step. Unreliable body landmarks could be used, for example, if it is not visible in one or more image frames or goes out of the field of view of the 2D digital camera 114, or in low light conditions. In this way, the method 200 prevents large errors in joint angle calculations if a body landmark is momentarily lost or tracked incorrectly (e.g. an arm leaving the camera frame). A confidence value that can be used is the 'min_pose_detection_confidence' variable output from the ML model 116. The threshold for the confidence value may be adjusted based on empirical evaluation and the specific requirements of the application. The present inventors have determined that a threshold of approximately 0.4 might be appropriate for scenarios characterised by generally lower confidence values output by the machine learning model 116 (for instance, in low-light environments or challenging poses) in order to avoid the removal of an large number of body landmarks. Conversely, in situations where a higher precision is important, a threshold of approximately 0.6 could be more suitable to ensure that only the most reliable body landmarks are taken into account.

[0151] If each selected body landmark has a confidence value greater than threshold, the method 200 proceeds to step S207 in which the aforementioned z co-ordinate data estimated by the ML model 116 is removed from each selected body landmark in the frame of image data. In terms of co-ordinate data, this leaves only data representing the x and y co-ordinates for each selected body landmark, i.e. the location of the selected body landmark in the image plane of the 2D digital camera.

[0152] At step S208 a 2D joint angle determination is made. A joint angle may be an angle between two segments defined by three selected body landmarks. For example, referring again to Figure 2, an example joint angle of the right elbow can be seen defined by the two lines joining body landmark numbers 12, 14 and 16. It is noted that the joint angles defined by the selected body landmarks may or may not correspond to the physical joints of the subject 102.

[0153] It will be apparent that the limbs of the 'stick person' shown in Figure 3 are a projection onto the 2D page which corresponds to the projection of the subject 102 onto the image plane of the 2D digital camera 114. Accordingly, the joint angle defined by the two lines joining body landmark numbers 12, 14 and 16 is a 2-dimensional angle which will probably not correspond to the actual joint angle of the subject 102 in 3 dimensions. For example, the body landmarks at the ends of the right arm and forearm of subject 102 may have different z co-ordinates (i.e. depths) relative to the image plane of the 2D digital camera 114. However, the present inventors have discovered that, whilst the z co-ordinate output from the ML model is not accurate and is only given relative to the hips of the subject 102, the x and y co-ordinates output from the ML model 116 are reasonably accurate. The inventors have found that the x and y co-ordinates can be used to further filter the data toP5642PC00-MH8NG

[0154] ensure to improve accuracy of a z co-ordinate estimation performed later in the method 200. This filtering may be performed by determining the 2D joint angle formed by the selected body landmarks, and then checking whether the calculated 2D joint angle is an outlier.

[0155] To determine each joint angle in the image frame, the computer-implemented device 100 forms two vectors. If the selected body landmarks are A, B and C, having co-ordinates (xA, YA), (XB, YB) and (xc, yc) respectively, the vectors are formed as:

[0156] ba = (xA-xB,yA-yB)

[0157] be = (xc— xB,yc— yB)

[0158] These two vectors lie in the image plane of the 2D digital camera 114. The 2D joint angle, 9, between the vectors is determined using:

[0159]

[0160] The 2D joint angle 9 is stored in the memory 112.

[0161] At step S209 the method 200 performs further data filtering on the image frames, in which further metrics about the calculated values of angle 9 are determined and used as a basis to remove body landmarks and / or image frames from further processing. A 'smoothed' 2D joint angle, 9smoothed, is determined with time to provide a stable reference for each new incoming 2D joint angle, 9new. 9smoothed may be determined as:

[0162]

[0163] where N is the total number of image frames processed so far. The incoming 2D joint angle, 9new, is then checked against 9smoothed: if 9new deviates from 9smoothed by more than a certain threshold (for example n standard deviations or a fixed numeric cutoff), 9new is flagged as an outlier. The choice of the number n of standard deviations depends on the preference for removing outlier 2D joint angles. Common values of n include:

[0164] • n = 2: identifies approximately the top and bottom 5% of data (assuming a roughly normal distribution).

[0165] • n = 3: identifies approximately the top and bottom 0.3% of data, making the method more conservative in discarding body landmarks.P5642PC00-MH8NG

[0166] In a clinical or ergonomic context, a somewhat stricter threshold (e.g., n=3) may be used to ensure that only the most extreme anomalies are discarded. However, if data is very noisy or if removing more image frames can be tolerated, n=2 might be preferred.

[0167] When any 2D joint angles is flagged as an outlier either the body landmarks corresponding to that 2D joint angle are removed from the process (if there are other 2D joint angles in the same frame which are not outliers, for example), or the entire image frame is removed. This helps to ensure that only body landmarks providing stable 2D joint angles within an expected range are passed to the next part of the process.

[0168] If the 2D joint angle is not flagged as an outlier, it is used to update the value of 9smoothed for the next image frame.

[0169] In this way the body landmarks and image frames output from step S209 are themselves 'smoothed' by removing brief spikes in incoming image frames, for example. This also helps to smooth the final output joint angles determined by the method 200 which may enhance downstream analysis, benefiting time-series methods such as the Kalman filter for z coordinate estimation. This smoothing ensures that minor fluctuations in angle measurements do not negatively affect depth or stability calculations.

[0170] The method 200 continues with reference to Figure 5. At step S210 image frames remaining after step S209 are subject to a time-series analysis to estimate a z co-ordinate for each selected body landmark in the image frame. The z co-ordinate represents the perpendicular distance between the image plane of the 2D digital camera 114 and the selected body landmark on the subject 102, i.e. a real-world measurement in metres (or other distance unit). To estimate z a Kalman filter may be used. For the first image frame z is estimated heuristically based on anthropometric ratios. For body landmarks on the torso (such as the nose, shoulders, or hips), the heuristic z-value is estimated using known body proportions. For limb body landmarks (such as elbows and knees), the approach is similar but with some adjustments. Body landmarks that are near the torso such as elbows and knees, the depth (i.e along the z-axis) can be quite small. This means that even if an elbow or knee is slightly in front of or behind the torso, it makes only a very small difference in relation to the overall depth of the body.

[0171] A normalised body model based on known average anthropometric ratios is assumed. See for example Dempster, W. T. (1955), "Space requirements of the seated operator", University of Michigan, WADC Technical Report and de Leva, P. (1996), "Adjustments to Zatsiorsky-Seluyanov's segment inertia parameters", Journal of Biomechanics, 29(9), 1223-1230. Using these ratios, an expected z position for the limbs relative to the torso is approximated. For example, the torso has an estimated z-value (from the heuristic initialisation), and the elbows and knees will be assigned a similar depth, but it will beP5642PC00-MH8NG

[0172] adjusted by a small scaling factor (described in greater detail below) derived from average anatomical data.

[0173] The starting value of z, zo, for the torso is determined as follows. As mentioned above, some key body landmarks are retained and not discarded in the first image frame (assuming they are not part of the body landmark data selection anyway). For example, and with reference to Figure 2, the body landmarks numbered 0 (nose) and 11 (left shoulder) or 12 (right shoulder) can be used as the key body landmarks. The starting value for the Kalman filter is calculated as:

[0174]

[0175] Hreai is an average head height, usually 13% of the person's height. The subject's height may be estimated in two ways. Firstly, the subject may enter their height into the app 111. This value, Htotai, typically measured in centimetres, is then used directly. From this, the average head height (Hreai) is calculated as a fixed proportion, approximately 13% of the total height (i.e., Hreai= 0.13 x Htotai). This approach leverages well-established anthropometric ratios.

[0176] Secondly, height may be estimated by calibration with a reference object in the image. For example, if the app 1111 is used by a subject at home, the app 111 may work without direct user input, and a calibration process will be used instead. This may involve capturing an image of the subject and comparing it with the reference average height derived from population data.

[0177] dpixeis is the aforementioned y distance in pixels between the nose body landmark and one of the left and right shoulder body landmarks. Since the focal length of the lens in the 2D digital camera 114 is not known (and may vary between different cameras and devices) k is a scaling constant. The scaling constant k may be found empirically. The present inventors have determined values for k as follows:

[0178] Smartphone Cameras: the empirical scaling constant k generally ranges from about 1.0 to 1.3;

[0179] Laptop Cameras: due to differences in sensor size and design, laptop cameras often yield a slightly higher k value, usually between 1.5 and 2.0.

[0180] These ranges are approximations and can differ based on the specific camera hardware, the distance to the subject, and various environmental conditions. The average standard value for the type of device (laptop, smartphone) can be determined empirically. However, even ifP5642PC00-MH8NG

[0181] the app 111 begins with a standard k value, the Kalman filter (or other time-series analysis) will progressively reach a relatively stable and accurate estimate for the z co-ordinate.

[0182] zo provides an estimate of the distance or depth of the shoulders away from the 2D digital camera 114. The Kalman filter updates this zo co-ordinate estimate for each subsequent frame.

[0183] To estimate the starting value of z for other body landmarks, it is assumed that each a similar depth as zo but adjusted by a small scaling factor 6:

[0184] z = z0x (1 + 5)

[0185] The scaling factor 6 is typically a small percentage, for example between ±3% and ±5% of the torso depth zo. Various correction factors 6 for some body landmarks are given in the following table:

[0186] "

[0187]

[0188] P5642PC00-MH8NG

[0189] Ankles ^Ankles ~ -0.05 tO +0.07 Ankles vary more in walking / running

[0190]

[0191] Each 6 represents the best average offset for a landmark relative to the torso's z-value, with a defined small range to accommodate individual variation and posture differences. It is possible to use statistical averaging to determine the mean and standard deviation of body landmarks relative to the torso for different subjects. For instance, if elbow body landmarks are found to be about 3% behind the torso in depth on average, 6eibows might be set to -0.03. The standard deviation, for example 0.02, would define a small range (±0.02) around this mean. Using the Kalman filter 6 may be refined over time, particularly if the user's posture changes significantly. The Kalman filter addresses any residual discrepancy between the chosen delta and real-world movements.

[0192] Given the frames per second of current digital cameras (e.g. 30 fps), the z co-ordinate estimates quickly approach the true value. The accuracy of the time-series analysis is improved by the earlier steps in the method to remove unreliable body landmarks that would result in determination of an incorrect joint angle, for example those with low confidence values and those which generate 2D joint angles that are outliers.

[0193] Embodiments of the invention are not limited to the use of a Kalman filter. Any technique suitable for estimating the z co-ordinate may be used, including various time-series forecasting methods. Exponential smoothing and ARIMA models are some examples. A depth inference algorithm may also be used to estimate the z co-ordinate of each body landmark.

[0194] Once the estimates for each z co-ordinate have been completed, each selected body landmark co-ordinate is converted at step S211 from 2 dimensions to 3 dimensions by supplementing the estimated z co-ordinate to the x and y co-ordinates.

[0195] At step S212 a 3D joint angle is determined for the selected body landmarks. Vectors for selected body landmarks are determined. If the selected body landmarks are A, B and C, having co-ordinates (xA, yA, zA), (xB, ye, zB) and (xc, yc, zc)

[0196] ba = (xA- xB,yA- yB,zA- zB~)

[0197] be = (xc- xB, yc- yB, zc- zB

[0198] The angle 0 between the vectors in three dimensions is then determined in the same way as described above for the angle 0 based on vectors ba and be in the x-y plane. Angle 0 is the 3D joint angle between the segments defined by the selected body landmarks.P5642PC00-MH8NG

[0199] At step S213 the 3D joint angle(s) can be visualized in various ways. For example, after analysis of all image frames in the digital video data, plots of 3D joint angle versus time can be displayed showing how the or each joint's angle changed throughout the session.

[0200] Figure 5A illustrates visual plots of the subject with ataxia performing various tasks: walking, sit-to-stand, and return to sit. All plots are graphs of image frame number (x-axis) versus hip angle flexibility. The upper plot was generated by the subject performing a walking task and ending with a sitting movement. The middle plot was generated by the same subject performing a stand-to-sit-to-stand task. The lower plot was generated by the subject performing a return-to-sit task, involving approaching a chair, turning around and sitting down.

[0201] Figure 5B illustrates example textual feedback provided by the app 111 on the basis of the results determined by the method 200 when checking posture of the subject (for example using the 2D digital camera to record the subject whilst working at a desk). By comparing calculated 3D joint angles with known ranges stored in the memory, the app 111 generates feedback such as "Left Shoulder (Flex) average angle 122.3° is above normal. Consider lowering your seat of adjusting posture."

[0202] At step S214 the or each 3D joint angle can be stored in the memory 112, for example in a CSV format. An advantage of the method 200 is that all digital video data is processed locally on the computer-implemented device 100 and all output data (i.e. 3D joint angles) is also stored locally. The user retains control of this data and can decide whether or not to share it.

[0203] It is also possible to provide substantially real-time feedback for the user. The body landmarks and segments joining them (also known as a 'pose skeleton', see Figure 3) can be overlaid live on the GUI 104 and / or in the playback of the digital video data image frames. This overlay helps the user see that their movement is being captured correctly (joint positions, etc.). Owing to the use of a separate execution thread for the ML model 116, the calculated 3D joint angle(s) may also be displayed substantially in real-time on the GUI 104 in various ways. For example, a time-averaged numerical value could be displayed and / or or an indication whether the 3D joint angle falls within a normal range. This enables the user to receive substantially real-time feedback, e.g. as they are performing a movement in front of the 2D digital camera 114.

[0204] Body Landmark Selection Mask

[0205] As described above, the user may select the body landmarks representing joint angles of interest. The body landmark selection mask is a sub-set or choice of the available body landmarks. The body landmark selection mask may be pre-packaged into common sets suitable for analysing posture and / or motion, as shown in Figures 6A - 6G. Additionally orP5642PC00-MH8NG

[0206] alternatively, the app 111 may permit the user to select their own combination of body landmarks.

[0207] Each body landmark selection mask may be directed to joint angles that help with diagnosing or assessing the user. For example, the body landmark selection mask may pick out joint angles for assessing gait, standing or sitting posture, balance, leg swing, sit-to-stand transition, etc. This is shown at step S215S214 in Figure 5.

[0208] Figure 6A shows a body landmark selection mask 600 referred to as DIGIMOTION™. Body selection mask 600 selects body landmarks representing the upper lower body joints: shoulders, elbows, wrists, hips, knees and ankles. The method 200 determines the 3D joint angles for this body landmark selection mask 600 a described above. At 602 a dual flexibility metric is determined using the 3D joint angles stored in memory which shows instantaneous vs. time-averaged joint behaviour: the raw data set (i.e. for each image frame) can be presented to provide a so-called 'flexibility' measure, i.e. the instantaneous 3D joint angle for each image frame, and a 'standard' set can be provided which could be a more stable set of 3D joint angles, such as the average value over a time interval (e.g. 1 s, 5 s, 10 s, etc.). Figure 6H shows some example plots of the flexibility measure and the standard measure for the left shoulder (plots 604 and 605) and for the right shoulder (plots 606 and 607) of a subject. Each plot is a graph of frame number (x-axis) versus joint angle in degrees (y-axis).

[0209] Figure 6B shows a body landmark selection mask 610 referred to as DIGIERGO™. Body selection mask 610 selects body landmarks representing posture: nose, right and left shoulders, right and left hips, right and left eyes, spine (a 'proxy' landmark for the spine may be determined by determining the average of multiple significant body landmarks, such as the left and right shoulders as well as the left and right hips, to estimate the central axis of the torso). The method 200 determines the 3D joint angles for this body landmark selection mask 610 as described above. At 612 the neck angle and head tilt are determined and at 614 an assessment of ergonomic posture is determined by comparing neck angle and head tilt to normative ranges. The results can be provided to the user in an interactive format on the GUI 104. For example, an interactive information panel can be overlaid onto a live camera or over the digital video data. The panel may comprise colour-coded feedback (e.g. green for angles within recommended ranges and red for deviations). Results may be stored in the memory 112 for future use. This enhances ergonomic assessments by enabling immediate, accessible posture analysis, together with long-term data logging for remote evaluation.

[0210] Figure 6C shows a body landmark selection mask 620 referred to as DIGIBALANCE™. Body selection mask 620 selects body landmarks representing lower body and core: right and left hips, right and left knees, right and left ankles, right and left feet, spine (determined asP5642PC00-MH8NG

[0211] described above). The method 200 determines the 3D joint angles for this body landmark selection mask 620 as described above, i.e. hips, knees and ankles as shown at 622.

[0212] At 624 the subject's centre of mass (COM) is determined in real-time using a weighted model (e.g. Winter's model) based on the selected body landmarks. However, rather than a simple geometric centre, a weighted average of key body segment positions is determined, accounting for the fact that different parts of the body contribute unequally to the overall mass. Each major body segment (or its representative body landmark) is detected and measured with a weight proportional to its share of total body mass. These weight factors are informed by biomechanical anthropometric data - for example, the torso (trunk) contributes roughly 50-55% of body weight, each thigh about 10%, and shanks around 5% in an average adult. Individual anatomical differences can be accommodated by adjusting these weights based on the user's profile, such as sex, height, or body type, which alters segment mass distribution. Using these weights, the COM position is computed as the weighted sum of the coordinates of all selected body landmarks divided by the total weight - effectively a mass-weighted centroid of all the tracked points. This provides a more accurate representation of the body's balance point than an unweighted centre.

[0213] Raw body landmark data can be noisy (due to slight detection errors or camera noise). The COM calculation may be stablised with filtering techniques. A Kalman filter may be used to leverage the data's time-series nature, predicting the COM movement based on previous motion and correcting that prediction with the new measurements, thus filtering out high-frequency jitter. In this way a COM path is determined that is much less noisy while still responsive to real changes as the subject 102 moves. This results in a stable, continuous estimation of the subject's centre of mass even as they move, accounting for measurement uncertainties. Additional smoothing (such as a moving average or one-euro filtering) may also be applied to further suppress transient spikes. The output is a robust real-time COM signal that accurately reflects the user's balance point and is not overly perturbed by momentary detection errors.

[0214] At 626 a so-called 'stability index' is determined. This comprises projecting the COM down to the ground plane and measuring its horizontal distance from the user's base of support. The x-z coordinates of the COM are used (essentially ignoring height for the projection) to compare this to the average position of the feet on the ground. The feet (usually using the ankle or foot landmarks for left and right) define the base of support area. When standing, the base is roughly the area between and around the feet. The midpoint between the left and right feet positions (i.e. the centre of the stance) is determined and then the Euclidean distance from this midpoint to the COM's ground projection is calculated. This distance is the stability index, representing how far the user's centre of mass is deviating from the centre of their support base.P5642PC00-MH8NG

[0215] A small stability index (COM near the middle of the feet) indicates a stable stance. A larger stability index means the COM is closer to the edge of the base, which corresponds to a more precarious balance. If the COM projection were to move beyond the area covered by the feet, the person would likely need to take a step or risk falling. In this way, the stability index provides a quantitative measure of how balanced the posture is at each moment. In some aspects, the stability index may be normalized through a biomechanical model for consistency. The stability index may be normalised to account for differences in individual foot span or stance width. For example, the app 111 may determine the horizontal displacement of the projected centre of mass (COM) relative to the midpoint between the feet. This distance may then be expressed as a percentage of half the stance width (or may be directly compared against a threshold based on the person's foot span). For instance, if a subject's feet are farther apart, their absolute COM displacement might be larger, but when normalised (divided by half the stance width), the index reflects a similar relative challenge to balance as for someone with a narrower stance. This normalisation ensures that the stability index is comparable across individuals, regardless of variations in foot placement or body size.

[0216] To refine accuracy, the system can incorporate known stability criteria. For example, in dynamic situations (like slight swaying or during movement), algorithms from research consider the extrapolated COM (which factors in velocity) relative to the base of support to judge stability. In dynamic situations, static COM measures are insufficient because they don't capture the effects of momentum. To improve accuracy, the app 111 may incorporate stability criteria based on the extrapolated centre of mass (XCOM), which accounts for COM velocity. XCOM may be calculated in the app 11 as:

[0217]

[0218] VCOM is the velocity of the centre of mass, and w0is a constant derived from gravitational acceleration and the effective height of the COM (w0= jg / h) where h is the effective COM height. The extrapolated COM is then projected onto the horizontal plane (x-z) and its position relative to the base of support compared to derive the stability index. This approach captures both the static position and dynamic motion (momentum) of the COM, providing a robust measure of balance regardless of individual differences in foot span or stance width. This allows the app 111 to predict where the COM is heading, rather than simply where it is.

[0219] The stability index is then determined by measuring the horizontal distance between the XCOM and the boundaries (or the midpoint) of the base of support— normalised, for example, as a percentage of half the stance width. This method provides a robust measure of balance by reflecting both the position and momentum of the COM, thereby offering a more accurate assessment of dynamic stability.P5642PC00-MH8NG

[0220] By tracking the stability index substantially continuously, the app 111 can quantify balance in real time, giving a measure of how "centred" the user is. This real-time stability index is useful for providing immediate feedback on posture and balance quality.

[0221] As the stability index is monitored (optionally with other posture metrics), visual cues may be overlaid on the live video feed shown on the GUI 104 to guide the user toward better balance.

[0222] Real-time error correction adjustments may be made to dynamically adjust sensitivity and cues based on the ongoing data. For example, if the environment or camera angle changes, the baseline "centre" position will be recalibrated to ensure the feedback remains accurate.

[0223] The COM and stability index data may be logged with timestamps. Time-series charts may be displayed on the GUI 104 to show how the stability index fluctuates, reinforcing the feedback (the user can correlate their movements with the changes on the chart in realtime). Cues may be provided, such as coloured overlays and / or sounds and / or graphs, inform the user immediately when their posture deviates and enable them to correct it straight away.

[0224] Figure 6D shows a body landmark selection mask 630 referred to as DIGISWING™. Body selection mask 630 selects body landmarks representing the lower body: right and left hips, right and left knees, right and left ankles. The method 200 determines the 3D joint angles for this body landmark selection mask 630 as described above. At 632 leg swing direction may be determined by comparing hip and ankle co-ordinates. Dynamics may be determined by the directional movement and horizontal displacement of leg landmarks, counting swing events, and determining asymmetry between left and right sides.

[0225] Figure 6E shows a body landmark selection mask 640 referred to as DIGIStS™. Body selection mask 640 selects body landmarks representing the core and lower body: right and left shoulders, right and left hips, and right and left knees. The method 200 determines the 3D joint angles for this body landmark selection mask 640 as described above. At 642 sit-to-stand transitions are detected in the digital video data by analysing changes in postural angles via hip, knee and torso angles. The computer-implemented device 100 may log the posture of the subject 102 m to record times sitting, standing and in transition to standing.

[0226] Figure 6F shows a body landmark selection mask 650 referred to as DIGIGAIT™. Body selection mask 650 selects body landmarks representing the core and lower body: right and left shoulders, right and left hips, and right and left knees. The method 200 determines the 3D joint angles for this body landmark selection mask 650 as described above. At 652 gait metrics are determined including step time, step length, stride length, walking speed, cadence and arm swing asymmetry.P5642PC00-MH8NG

[0227] Figure 6G shows a body landmark selection mask 650 called DIGIMOVE™. Body selection mask 660 selects body landmarks representing the core and lower body: right and left shoulders, right and left hips, and right and left knees. The method 200 determines the 3D joint angles for this body landmark selection mask 660 as described above. At 662 general movement metrics are determined including head-to-ankle ratio and overall mobility.

[0228] The head-to-ankle ratio and overall mobility may be determined by analysing the 3D spatial positions of key body landmarks over time.

[0229] For the head-to-ankle ratio we first extract the 3D coordinates of the head (for example by using the nose body landmark or an estimated top-of-head body landmark) and of the ankles (for example by using the average of the left and right ankle body landmarks).

[0230] The vertical distance between these points is determined as:

[0231]

[0232] This distance is then normalised by the subject's estimated height (or a reference value derived from a stable set of landmarks such as the torso). This ratio provides an index of overall body proportion and posture:

[0233] Distancehead

[0234] Head — to — ankle Ratio = -to-ankle

[0235] Estimated body height

[0236] For overall mobility, various indicators may be determined. For example, a Range of Motion (ROM) may be determined. For example, for each joint of interest, the app 111 may determine the maximum and minimum angles over a defined time interval. The difference (such as the range) is a direct indicator of mobility.

[0237] A Dynamic Variability may be determined. The app 11 may determine time-based metrics such as the mean angular velocity or acceleration of joints, which indicate how rapidly and smoothly the joints are moving.

[0238] A Composite Index may be determined. These measures may be combined— often using statistical models or weighted averages— to form an overall mobility index that reflects both the extent and the quality of movement across the body.

[0239] User Recommendations

[0240] In some embodiments the app 111 may be adapted to generate textual and / or graphical feedback that evaluates the joint flexibility of the subject 102. Once data is stored following the method 200, a "Get Feedback" function may compute the average angle at each joint and then compare that against known healthy ranges for that joint. Normal angle ranges (for shoulders, elbows, etc.) may be built into the app 111 as a dictionary, enabling the appP5642PC00-MH8NG

[0241] 111 to quickly check whether the average joint angle within a healthy range or otherwise. If a joint's average angle is below the normal range, the app 111 may generate an indication that this could indicate limited flexibility and provide recommended exercises to improve it. All these feedback messages may be gathered and displayed to the user in a text box in the GUI 104. This aspect turns raw angle data into actionable insight for the user and automates what a human coach or clinician might do.

[0242] A user interface for the software may support straightforward data management. Buttons may be provided enabling the user to save the angle data to CSV (which includes timestamps and all angle values per frame) and to open the data folder directly. This makes reviewing results or sharing them straightforward.

[0243] Remote Analysis

[0244] Figure 7 illustrates an embodiment of a system 700 in which computer-implemented device 100 may connect to a software-as-a-service (Saas) function in a cloud computing platform 702, shown as portal 704. In some embodiments the app 111 may be installed on the computer-implemented device 100. The user may choose whether to share data (i.e. 3D joint angles and / or suggestions determined by the method 200) with the portal 704. A remote user, such as a clinician, may access the portal 704 via any suitable device such as laptop 706, smartphone 708 and desktop computer 710, to view / download data shared by the user of the computer-implemented device.

[0245] In other embodiments, the computer-implemented device 100 may not comprise the app 111. Instead the functionality of the app 111 may be performed by the cloud computing platform 702. The user of the computer-implemented device 100 may upload digital video data (either pre-recorded or 'live' as a video call) and the method 200 performed by the cloud computing platform 702. The calculated joint angles may be shared with the user of the computer-implemented device 100 and / or with the remote user of devices 706, 708, 710.

[0246] In other embodiments, any of the devices 706, 708, 710 may comprise the app 111. The remote user may then receive digital video data of the user of the computer-implemented device 100 performing exercises or other movements which are then analysed on any of devices 706, 708, 710. The digital video data may be 'live', e.g. received in a video call over the Internet, or may be pre-recorded. In this way the remote user (e.g. physiotherapist, doctor, etc.) may use the data and results generated by the app 111 to assist the user. This may be useful in 'telemedicine' when a user receives assistance from a medical professional remotely, for example via a video call.P5642PC00-MH8NG

[0247] In some embodiments the app 111 provides a full pipeline from capture to analysis to report, all in one tool. Users may receive real-time visualisation (e.g. skeleton overlay) and post-session analysis (graphs and written feedback).

[0248] While the invention has been disclosed in conjunction with a description of certain embodiments, including those that are currently believed to be useful embodiments, the detailed description is intended to be illustrative and should not be understood to limit the scope of the present disclosure. As would be understood by one of ordinary skill in the art, embodiments other than those described in detail herein are encompassed by the present invention. Modifications and variations of the described embodiments may be made without departing from the spirit and scope of the invention.

[0249] It will further be understood that any of the ranges, values, properties, or characteristics given for any single component of the present disclosure can be used interchangeably with any ranges, values, properties, or characteristics given for any of the other components of the disclosure, where compatible, to form an embodiment having defined values for each of the components, as given herein throughout. Further, ranges provided for a genus or a category can also be applied to species within the genus or members of the category unless otherwise noted.

Claims

P5642PC00-MH8NGCLAIMS1. A computer-implemented device for determining joint angles of a subject, which device comprises:a digital camera adapted to output digital video data comprising frames of image data in two-dimensions (2D) in an image plane of the digital camera;at least one machine learning model; anda memory for storing said digital video data in which the frames of image data contain at least one joint of the subject;the computer-implemented device adapted, for each frame of image data, to:determine using said at least one machine learning model a plurality of body landmarks of the subject, each body landmark comprising 2- dimensional (2D) co-ordinate data representing the co-ordinates of the body landmark in a first dimension and a second dimension within the image plane of the digital camera;perform a time-dependent analysis of said plurality of body landmarks to estimate a third-dimension co-ordinate for each body landmark in said frames of image data, said third dimension mutually perpendicular to said first and second dimensions;use said third dimension co-ordinates to convert said 2D co-ordinate data into three-dimensional (3D) co-ordinate data for each body landmark; use said 3D co-ordinate data of said plurality of said body landmarks to determine a 3D joint angle of said at least one joint of the subject within respective frames of image data; andoutput data representing said 3D joint angle.

2. A computer-implemented device as claimed in claim 1, further adapted to identify and discard from said plurality of body landmarks any unreliable body landmark in said plurality of body landmarks.

3. A computer-implemented device as claimed in claim 2, wherein said unreliable body landmark comprises a body landmark that would result in determining a joint angle outside an expected range and / or a body landmark that is missing or occluded in said frame of image data.

4. A computer-implemented device as claimed in claim 2 or 3, wherein said machine learning model is adapted to output a confidence value with each body landmark detected, the confidence value representing how likely it is that the body landmark has been correctly identified, and wherein to identify and discard an unreliable body landmark said computer-implemented device is further adapted to:P5642PC00-MH8NGdetermine whether said confidence value greater or less than a threshold, and to discard said body landmark as unreliable if said confidence value is lower than said threshold.

5. A computer-implemented device as claimed in claim 2, 3 or 4, wherein to identify and discard an unreliable body landmark said computer-implemented device is further adapted to:use said 2D co-ordinate data for a plurality of said body landmarks to estimate a 2D joint angle of said at least one joint of the subject within respective frames of image data, each said 2D joint angle lying in said image plane of the digital camera.

6. A computer-implemented device as claimed in claim 5, wherein to identify and discard an unreliable body landmark said computer-implemented device is further adapted to:determine whether said 2D joint angle is an outlier compared to previous calculations of said 2D joint angle in said digital video data.

7. A computer-implemented device as claimed in claim 5 or 6, wherein to identify and discard an unreliable body landmark said computer-implemented device is further adapted to:perform a statistical analysis of said 2D joint angle, such as determining whether said joint angle is within n standard deviations of 2D joint angles determined previously in said digital video data.

8. A computer-implemented device as claimed in any preceding claim, wherein to perform said time-dependent analysis said computer-implemented device is further adapted to:use a Kalman filter, exponential smoothing or an Auto Regressive Integrated Moving Average (ARIMA) to estimate said third-dimension coordinate for each frame of image data.

9. A computer-implemented device as claimed in claim 8, further adapted to determine a first expected value of said third-dimension co-ordinate using anthropometric data and to input said first expected value into said time-dependent analysis.

10. A computer-implemented device as claimed in claim 9, wherein said first expected value relates to a body landmark on the torso of the subject, the computer-implemented device adapted to adjust said first expected value of said body landmark on the torso by a scaling factor, and to use said adjusted first expected value to determine a first expected value of said third dimension for another body landmark, such as a body landmark on a limb of the subject.P5642PC00-MH8NG11. A computer-implemented device as claimed in any preceding claim, wherein said machine learning model is adapted to output a third dimension estimate of each body landmark and wherein said computer-implemented device is further adapted to discard each said third-dimension estimate and to use only said 2-dimensional (2D) co-ordinate data output by said machine learning model.

12. A computer-implemented device as claimed in any preceding claim, wherein said computer-implemented device comprises a processor and is further adapted to use a separate thread of execution within said processor to determine said plurality of body landmarks with said machine learning model.

13. A computer-implemented device as claimed in any preceding claim, further adapted to display on a graphical user interface any body landmarks representing a joint angle identified in said frame of image data and / or any indication based on said output 3D joint angle, before processing a next frame of image data.

14. A computer-implemented device as claimed in any preceding claim, further adapted to store each output 3D joint angle, optionally with other data such a timestamp of the output 3D joint angle.

15. A computer-implemented device as claimed in any preceding claim, further adapted to read a body landmark selection mask representing at least one joint angle of the subject, and to discard body landmarks output by the machine learning model that are not identified in said body landmark selection mask.

16. A computer-implemented device as claimed in claim 14, wherein said body landmark selection mask identifies body landmarks corresponding to one or more joint angle useful for analysing posture and / or movement of the subject.

17. A computer-implemented device as claimed in any preceding claim, wherein said body landmarks represent upper and lower body joints of the subject, the computer-implemented device further adapted to determine an average value of said 3D joint angles over a period of time, and to store said average values of said 3D joint angles alongside values of said 3D joint angle determined for each frame of image data.

18. A computer-implemented device as claimed in any preceding claim, wherein said body landmarks represent a posture parameter of the subject, the computer-implemented device further adapted to overlay on said digital video data an indication when said posture parameter is within a normal range.

19. A computer-implemented device as claimed in claim 18, wherein said computer-implemented device is further adapted to overlay said indication substantially in real-time.P5642PC00-MH8NG20. A computer-implemented device as claimed in claim 18 or 19, wherein said body landmarks comprise the nose, left and right shoulder and left and right hips, and optionally left and right eyes, the computer-implemented device further adapted to use said body landmarks to determine neck angle and head tilt, and to compare said neck angle and head tilt to said normal range.

21. A computer-implemented device as claimed in any preceding claim, wherein said body landmarks represent a lower body and a core of the subject, the computer-implemented device further adapted to use said body landmarks to determine a centre of mass of the subject with each frame of image data.

22. A computer-implemented device as claimed in claim 21, wherein the computer-implemented device is further adapted to determine said centre of mass using a weighted average of body segments corresponding to said body landmarks.

23. A computer-implemented device as claimed in claim 21 or 22, wherein the computer-implemented device is further adapted to perform a time-series analysis of said centre of mass.

24. A computer-implemented device as claimed in claim 21, 22 or 23, wherein the computer-implemented device is further adapted to determine a stability index for the subject by determining a horizontal distance between a midpoint of a line joining body landmarks of the feet of the subject and the centre of mass projected onto the ground.

25. A computer-implemented device as claimed in claim 21, 22, 23 or 24, wherein the computer-implemented device is further adapted to track said stability index substantially continuously.

26. A computer-implemented device as claimed in any preceding claim, wherein said body landmarks represent a lower body of the subject including hip and ankle, the computer-implemented device further adapted to use said body landmarks to determine leg swing direction by comparing x co-ordinates of the hip and ankle.

27. A computer-implemented device as claimed in any preceding claim, wherein said body landmarks represent a core and a lower body of the subject, and wherein the computer-implemented device is further adapted to detect sit-to-stand transitions by determining changes between frames of image data in the 3D joint angles of the core and lower body.

28. A computer-implemented device as claimed in any preceding claim, wherein said body landmarks represent a lower body of the subject, and wherein the computer-P5642PC00-MH8NGimplemented device is further adapted to determine gait metrics of the subject during a walking motion, such as step length, stride length, cadence and walking speed.

29. A computer-implemented device as claimed in any preceding claim, wherein said body landmarks represent substantially a full body of the subject, and wherein the computer-implemented device is further adapted to determine a head to ankle ratio.

30. A computer-implemented device as claimed in any preceding claim, wherein said memory stores a range of normal values for each said 3D joint angle, and wherein the computer-implemented device is further adapted to determine whether said 3D joint angle lies within said range.

31. A computer-implemented device as claimed in any preceding claim, in the form of a personal computer such as a smartphone, tablet, gaming device, laptop or desktop computer.

32. A computer-implemented device as claimed in any preceding claim, wherein said 2D digital camera is integrated into a body of the computer-implemented device.

33. A computer-implemented method for determining joint angles of a subject, which method comprises the steps of:receiving from a digital camera digital video data comprising frames of image data in two-dimensions (2D) in an image plane of the digital camera and containing at least one joint of the subject;storing said digital video data;for each frame of image data:determining using at least one machine learning model a plurality of body landmarks of the subject, each body landmark comprising 2- dimensional (2D) co-ordinate data representing the co-ordinates of the body landmark in a first dimension and a second dimension within the image plane of the digital camera;identifying and discarding from said plurality of body landmarks any unreliable body landmark;performing a time-dependent analysis of said plurality of body landmarks to estimate a third-dimension co-ordinate for each body landmark in said frames of image data, said third dimension mutually perpendicular to said first and second dimensions;using said third dimension co-ordinates to convert said 2D co-ordinate data into three-dimensional (3D) co-ordinate data for each body landmark;P5642PC00-MH8NGusing said 3D co-ordinate data of said plurality of said body landmarks to determine a 3D joint angle of said at least one joint of the subject within respective frames of image data; andoutputting data representing said 3D joint angle.

34. A computer program product comprising a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method of claim 33, and optionally wherein the computer program product comprises an application downloadable from a server.

Images