Methods and apparatus for refining specific virtual keypoint locations and for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles

Abstract

According to embodiments, a method and apparatus for refining specific virtual keypoint locations of an upper limb of a primate subject is provided. The method includes calculating a wrist joint center and elbow joint center; refining the wrist joint center based on the elbow joint center, a median forearm length, and a directional vector associated with the elbow and wrist joint centers; and based on the refined wrist joint center, a rotated unit vector and median wrist radius, determining refined specific virtual keypoint locations. The rotated unit vector is associated with metacarpophalangeal joint positions marking fingers of the subject, the directional vector and a forearm cone formed by the elbow joint center, refined wrist joint center, and some upper-limb virtual keypoint locations. Further embodiments provide a method / system for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of the subject using refined specific virtual keypoint locations.

Description

METHODS AND APPARATUS FOR REFINING SPECIFIC VIRTUAL KEYPOINT LOCATIONS AND FOR ENHANCING MEASUREMENTS OF DISTAL UPPERLIMB JOINT ANGLES AND / OR BODY SEGMENT ROLLING ANGLESCross-Reference To Related Application

[0001] This application claims the benefit of priority of Singapore patent application No. 10202403483X, filed 8 November 2024, the content of it being hereby incorporated by reference in its entirety for all purposes.Technical Field

[0002] Various embodiments relate to a method and an apparatus for refining specific virtual keypoint locations of an upper limb of a primate subject for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of the primate subject. Various embodiments further relate to a method and an apparatus for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of a primate subject.Background

[0003] Assessing and tracking the progress of patients in rehabilitation can be challenging for therapists, especially when it comes to measuring the recovery of coordination in the distal part of the upper limb. Complex movements, such as forearm pronation / supination, wrist joint angles, and finger joint angles, are essential for daily activities and quality of life. However, manually measuring these parameters is a time-consuming process and can be prone to subjective interpretation.

[0004] While optical-based 3-dimensional (3D) motion capture (mocap) systems have revolutionized motion analysis by enabling full-body tracking, such systems that are commonly used in entertainment applications prioritize overall body movement often fall short in precise tracking and / or accurately capturing distal upper-limb joint angles Thislimitation may be sufficient and acceptable for gaming, animation, and interactive art purposes, but is especially problematic in medical applications and rehabilitation, where the precise measurement of these angles is essential for customizing interventions and monitoring progress effectively and / or range of motion precisely.

[0005] Thus, there is a need for a method and system for accurately tracking distal upperlimb joint angles and performing automated and quantitative measurement of forearm pronation / supination, wrist joint angles, and finger joint angles, thereby addressing at least the problems mentioned above.Summary

[0006] According to an embodiment, a method of refining specific virtual keypoint locations of an upper limb of a primate subject for enhancing measurements of distal upperlimb joint angles and / or body segment rolling angles of the primate subject is provided. The method includes determining a median forearm length of the primate subject, a median wrist radius, and 3 -dimensional (3D) full-body virtual keypoint locations of the primate subject including upper-limb virtual keypoint locations and metacarpophalangeal (MCP) joint positions marking fingers of the primate subject; calculating, based on the upper-limb virtual keypoint locations, a wrist joint center and an elbow joint center; refining the wrist joint center based on the elbow joint center, the median forearm length, and a directional vector associated with the elbow joint center and the wrist joint center, to obtain a refined wrist joint center; determining a rotated unit vector; and based on the refined wrist joint center, the rotated unit vector, and the median wrist radius, determining refined specific virtual keypoint locations from the upper-limb virtual keypoint locations. The rotated unit vector is associated with the MCP joint positions, the directional vector and a forearm cone. The forearm cone is formed by the elbow joint center, the refined wrist joint center, and a selection of the upper-limb virtual keypoint locations.

[0007] According to an embodiment, a method for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of a primate subject is provided. The method may include determining refined specific virtual keypoint locations using a method of refining specific virtual keypoint locations of an upper limb of the primatesubject, wherein the refined specific virtual keypoint locations include a refined ulnar styloid process marker (USPnew) and a refined radius styloid process marker (RSPnew); based on the RSPnew, the refined wrist joint center, and a half metacarpophalangeal marker (HMC), calculating an orientation of a hand segment to determine an x-axis, a y-axis and a z-axis of the hand segment; based on the RSPnew, the refined wrist joint center, and the elbow joint center, calculating an orientation of a forearm segment to determine an x-axis, a y-axis and a z-axis of the forearm segment; and subtracting 90 degrees from a measurement angle between one of the x-axis, y-axis or z-axis of the hand segment and one of the x-axis, y-axis or z-axis of the forearm segment to enhance measurement of one of the distal upper-limb joint angles. The measurement angle forms an anatomically meaningful relationship between the hand segment and the forearm segment.

[0008] According to an embodiment, an apparatus for refining specific virtual keypoint locations of an upper limb of a primate subject for enhancing measurements of distal upperlimb joint angles and / or body segment rolling angles of the primate subject is provided. The apparatus includes a computer configured to calculate, based on upper-limb virtual keypoint locations from 3D full-body virtual keypoint locations of the primate subject, a wrist joint center and an elbow joint center; refine the wrist joint center based on the elbow joint center, a median forearm length, and a directional vector, to obtain a refined wrist joint center; determine a rotated unit vector based on metacarpophalangeal (MCP) joint positions from the 3D full-body virtual keypoint locations of the primate subject, the directional vector and a forearm cone; and determine refined specific virtual keypoint locations from the upper- limb virtual keypoint locations based on the refined wrist joint center, the rotated unit vector and a median wrist radius. The MCP joint positions are for marking fingers of the primate subject. The directional vector is associated with the elbow joint center and the wrist joint center The forearm cone is formable by the elbow joint center, the refined wrist joint center, and a selection of the upper- limb virtual keypoint locations.

[0009] According to an embodiment, a system for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of a primate subject is provided. The system includes an apparatus for refining specific virtual keypoint locations according to an embodiment configured to determine refined specific virtual keypoint locationsincluding a refined ulnar styloid process marker (USPnew) and a refined radius styloid process marker (RSPnew). The computer is further configured to calculate an orientation of a hand segment to determine an x-axis, a y-axis and a z-axis of the hand segment based on the RSPnew, the refined wrist joint center, and a half metacarpophalangeal marker (HMC); calculate an orientation of a forearm segment to determine an x-axis, a y-axis and a z-axis of the forearm segment based on the RSPnew, the refined wrist joint center, and the elbow joint center; and subtract 90 degrees from a measurement angle between one of the x-axis, y-axis or z-axis of the hand segment and one of the x-axis, y-axis or z-axis of the forearm segment to enhance measurement of one of the distal upper-limb joint angles. The measurement angle forms an anatomically meaningful relationship between the hand segment and the forearm segment.Brief Description of the Drawings

[0010] In the drawings, like reference characters generally refer to like parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which:

[0011] FIG. 1A shows a flow chart illustrating a method of refining specific virtual keypoint locations of an upper limb of a primate subject for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of the primate subject, according to various embodiments.

[0012] FIG. IB shows a schematic view of an apparatus for refining specific virtual keypoint locations of an upper limb of a primate subject for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of the primate subject, according to various embodiments.

[0013] FIG. 2A shows a flow chart illustrating a method for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of a primate subject, according to various embodiments.

[0014] FIG. 2B shows a schematic view of a system for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of a primate subject, according to various embodiments.

[0015] FIG. 3 shows a schematic representation illustrating the front and back skeleton views with marker locations produced by the full-body markerless motion capture system, according to an example.

[0016] FIG. 4 A shows a tracking display (left) of a right forearm of a primate subject (right), where the right forearm is in a neutral position, according to an example.

[0017] FIG. 4B shows a tracking display (left) of the right forearm of the primate subject (right) of FIG. 4 A, where the right forearm is pronated, according to an example.

[0018] FIG 5 shows a graph illustrating plots comparing joint angle sequences for wrist extension / flexion using three different methods, according to an example.

[0019] FIG. 6 shows a graph illustrating plots comparing joint angle sequences for wrist radial / ulnar deviation using the three different methods, according to an example.

[0020] FIG. 7 shows a graph illustrating plots comparing joint angle sequences for forearm supination / pronation using the three different methods, according to an example.

[0021] FIGS. 8A and 8B show scatter plots of markerless measurement, without hand tracking fusion and with hand tracking fusion, respectively, against marker-based measurement (ground truth) for wrist flexion / extension angle.

[0022] FIGS. 9A and 9B show scatter plots of markerless measurement, without hand tracking fusion and with hand tracking fusion, respectively, against marker-based measurement (ground truth) for wrist radial / ulnar deviation angle.

[0023] FIGS. 10A and 10B show scatter plots of markerless measurement, without hand tracking fusion and with hand tracking fusion, respectively, against marker-based measurement (ground truth) for wrist forearm pronation / supi nation angle.Detailed Description

[0024] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled inthe art to practice the invention. Other embodiments may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0025] Embodiments described in the context of one of the methods or devices are analogously valid for the other methods or devices. Similarly, embodiments described in the context of a method are analogously valid for a device, and vice versa.

[0026] Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0027] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0028] In the context of various embodiments, the phrase “at least substantially” may include “exactly” and a reasonable variance

[0029] In the context of various embodiments, the term “about” as applied to a numeric value encompasses the exact value and a reasonable variance.

[0030] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0031] As used herein, the phrase of the form of “at least one of A or B” may include A or B or both A and B. Correspondingly, the phrase of the form of “at least one of A or B or C”, or including further listed items, may include any and all combinations of one or more of the associated listed items.

[0032] As used herein, the expression “configured to” may mean “constructed to” or “arranged to”.

[0033] Various embodiments may provide a fusion of full-body and hand markerless mocap (motion capture) to enhance upper-limb tracking. This fusion, involving the use of hand-tracking data, builds upon the foundation of markerless 3D full-body human motioncapture technology. The gap in existing systems may be addressed by extending the capabilities of this technology, with integration of advanced algorithms and techniques, to accurately and stably track distal upper-limb joint angles such as forearm pronation / supi nation and wrist joint angles.

[0034] FIG. 1A shows a flow chart illustrating a method 100 of refining specific virtual keypoint locations of an upper limb of a primate subject for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of the primate subject, according to various embodiments. As seen in FIG. 1A, at Step 102, a median forearm length of the primate subject, a median wrist radius, and 3-dimensional (3D) full-body virtual keypoint locations of the primate subject are determined. The 3D full-body virtual keypoint locations include upper-limb virtual keypoint locations and metacarpophalangeal (MCP) joint positions marking fingers of the primate subject. Each MCP joint position is marking each of the index, middle, ring and little (pinkie) fingers, respectively. At Step 104, a wrist joint center and an elbow joint center are calculated based on the upper-limb virtual keypoint locations. At Step 106, the wrist joint center is refined based on the elbow joint center, the median forearm length, and a directional vector, to obtain a refined wrist joint center. The directional vector is associated with the elbow joint center and the wrist joint center. At Step 108, a rotated unit vector is determined The rotated unit vector is associated with the MCP joint positions, the directional vector and a forearm cone. Specifically, the rotated unit vector is a unit vector associated with forearm rolling angle. The forearm cone is formed by the elbow joint center, the refined wrist joint center, and a selection (or some) of the upper-limb virtual keypoint locations. At Step 110, refined specific virtual keypoint locations from the upper-limb virtual keypoint locations are determined based on the refined wrist joint center, the rotated unit vector, and the median wrist radius.

[0035] In other words, the method 100 is a special data-driven method that focuses specifically on hand tracking. Every joint on the hand is located in 3D independently from 3D body tracking. The method 100 provides a technical solution to address the problem of data-driven full-body tracking method having a low ability to utilize the hand features to determine the orientation of the forearm along the rolling axis as the hand features arerelatively small compared to the whole human body. The method 100 is adapted to humans or other primates with similar anatomy.

[0036] In various embodiments, the median forearm length may be determined from a plurality of forearm lengths. The plurality of forearm lengths may include a forearm length being a distance between the elbow joint center and the wrist joint center, and one or more other forearm lengths calculated from one or more recorded trajectories of full-body markerless tracking of the primate subject, each of the one or more other forearm lengths being a distance between an elbow joint center and a wrist joint center, both of the elbow joint center and the wrist joint center derived from each of the one or more recorded trajectories. Use of the median forearm length may ensure robustness. The directional vector may be pointing from the elbow joint center to the wrist joint center (e g. see Equation (4) that will be described later below). The refined wrist joint center may be calculated by normalizing the directional vector to obtain a unit vector, and offsetting the elbow joint center with the unit vector scaled by the median forearm length along a direction of the unit vector (e.g. see Equation (3) that will be described later below).

[0037] The upper-limb virtual keypoint locations may include an ulnar styloid process marker (USP), a radius styloid process marker (RSP), a medial epi-condyle marker (HME), and a lateral epicondyle marker (HLE) These markers may be physical markers or virtual markers or a combination of physical and virtual markers.

[0038] Calculating the wrist joint center at Step 104 may include determining a midpoint of the RSP and the USP and calculating the elbow joint center may include determining a midpoint of the PILE and the HME (e.g. see Equations (1) and (2) that will be described later below).

[0039] In various embodiments, the method 100 may further include forming the forearm cone. The forearm cone may include a tip defined by the elbow joint center; and a circular base defined by the refined wrist joint center, the RSP and the USP. The circular base may have a radius being half a distance between the RSP and the USP.

[0040] The rotated unit vector may be determined at Step 108 by obtaining a direction vector pointing from one of the MCPjoint positions to a midpoint of a pair of other adjacent MCP joint positions, projecting the direction vector to a plane of the circular base of the forearm cone to obtain a projected vector subsequently normalizing the projected vector toobtain a normalized vector (or interchangeably referred to another unit vector), and rotating the normalized vector with a constant amount of offset angle around a core axis of the forearm cone in a pronation or supination direction to obtain the rotated unit vector.

[0041] The constant amount of offset angle may be obtained from a set of healthy subjects as follows. For example, for each healthy subject performing forearm pronation / supination task, the MCPs are tracked to determine the directional vector, which is projected to the cone base and normalized to obtain vector R* (not yet rotated by the offset). In the same frame, there is another directional vector G pointing from RSP to USP (with real physical markers captured by a marker-based motion capture system in the same reference frame). The directional vector G is also projected to the cone base and normalized to obtain a ground-truth vector G* As R* and G* are on the same plane of the cone base, it may be determined for how many degrees R* needs to rotate around the core axis of the cone for R* to match with G* in that one specific frame. With sufficient frames from one healthy subject, the median or mean of the offset samples can be taken from one healthy subject to represent that healthy subject. With adequate number of healthy subjects (1 offset sample per healthy subject), the median or mean of the offsets may be taken from the healthy population to be used as the constant amount of offset angle.

[0042] The median wrist radius may be determined from a plurality of wrist radius values. The plurality of wrist radius values may include the radius (of the circular base) being half the distance between the RSP and the USP, and one or more other wrist radius values calculated from one or more recorded trajectories of full-body markerless tracking of the primate subject, each of the one or more other wrist radius values being half a distance between a RSP and a USP, both derived from each of the one or more recorded trajectories. The one or more recorded trajectories are based on the full records of the same primate subject, and any suitable full-body markerless tracking techniques may be used to obtain the records. The records may also be taken at different points in time, i.e. time-independent.

[0043] Determining the refined specific virtual keypoint locations at Step 110 may include offsetting the refined wrist joint center with the rotated unit vector scaled by the median wrist radius along a direction of the rotated unit vector to obtain a refined ulnar styloid process marker (USPnew) and along an opposite direction of the rotated unit vector to obtaina refined radius styloid process marker (RSPnew) (e.g. see Equations (5) and (6) that will be described later below).

[0044] In some examples, the MCP joint positions may be obtained or determined from any available sources Tn other words, 2D images of the upper limbs of the primate subject, captured in videos, may be triangulated using known techniques to obtain 3D MCP joint positions.

[0045] In other examples, the MCP joint positions may be obtained from the 3D full-body virtual keypoint locations that may be generated (e.g. at Step 102) by an inference method. The inference method may include based on the primate subject captured by a plurality of colour video cameras as series of 2D images, for each 2D image captured by each colour video camera, generating, using a trained machine learning model, a 2D bounding box; for each 2D image, generating, by the trained machine learning model, a plurality of heatmaps with scores of confidence. Each heatmap may be for 2D localization of a keypoint of the primate subject. The inference method may further include for each heatmap, selecting a pixel with the highest score of confidence, and associating the selected pixel to the keypoint, thereby determining the 2D location of the keypoint; and based on the series of 2D images captured by the plurality of colour video cameras, triangulating the respective determined 2D locations to predict a sequence of 3D locations of the keypoint, thereby inferring the 3D full-body virtual keypoint locations. For each heatmap, the scores of confidence may be indicative of probability of having the associated keypoint in different 2D locations in the generated 2D bounding box.

[0046] Triangulating the respective determined 2D locations may include performing strategic triangulation of one trajectory of the keypoint at a time.

[0047] In these other examples, the trained machine learning model may be trained using at least a training dataset generated by a generation method. The generation method may include obtaining at least one augmented dataset comprising projected-marker-based annotated images of a plurality of test subjects; and sampling from the at least one augmented dataset and at least one in-the-wild dataset based on a sampling ratio to generate the training dataset. The at least one in-the-wild dataset may include manually annotated images of random subjects under unrestricted conditions. The projected-marker-based annotated images of the plurality of test subjects may include 2D marker-based keypointlocations, augmented keypoints and projected-marker-based bounding boxes. The manually annotated images of the random subjects may include 2D manually-annotated keypoint locations and manually -annotated bounding boxes. One or more of the augmented keypoints respectively may coincide with one or more of the 2D manually-annotated keypoint locations.

[0048] In the generation method, obtaining the at least one augmented dataset may include based on a plurality of physical markers captured by an optical marker-based motion capture system, each as a captured 3D trajectory, wherein each physical marker may be placed on a bone landmark or a keypoint of each of the plurality of test subjects, and the plurality of test subjects may be substantially simultaneously captured by the plurality of colour video cameras over a period of time as sequences of 2D images; for each physical marker, identifying the captured 3D trajectory with a marker label representative of the bone landmark or keypoint on which the physical marker may be placed; obtaining augmented markers; determining an augmented 3D trajectory with an augmented label representative of each augmented marker; for each physical marker, projecting the captured 3D trajectory and for each augmented marker, projecting the augmented 3D trajectory to each of the 2D images to determine a 2D location in each 2D image; for each physical marker and for each augmented marker, based on the respective 2D locations in the sequences of 2D images and an exposure-related time of the plurality of colour video cameras, interpolating a 3D position for each of the 2D images; for each 2D image, based on the respective interpolated 3D positions of the plurality of physical markers and the augmented markers, and an extended volume, generating each projected-marker-based bounding box around each test subject; and generating the augmented dataset including at least one 2D image selected from the sequences of 2D images, the determined 2D location of each physical marker in the selected at least one 2D image, the determined 2D location of each augmented marker in the selected at least one 2D image, and the projected-marker-based bounding boxes for the selected at least one 2D image.

[0049] The extended volume may be derived from two or more of the physical markers and / or the augmented markers having an anatomical, functional and / or structural relationship with one another. Each augmented marker may be calculated from positions of two or more of the physical markers placed on each test subject. For each physicalmarker, the marker label may be arranged to be propagated with each determined 2D location such that in the augmented dataset, each determined 2D location of each physical marker may contain the corresponding marker label. For each augmented marker, the augmented label may be arranged to be propagated with each determined 2D location such that in the augmented dataset, each determined 2D location of each augmented marker may contain the corresponding augmented label. The 2D marker-based keypoint locations may include the determined 2D locations of the plurality of physical markers. The augmented keypoints may include the determined 2D locations of the augmented markers.

[0050] FIG. IB shows a schematic view of an apparatus 120 for refining specific virtual keypoint locations of an upper limb of a primate subject for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of the primate subject, according to various embodiments.

[0051] Essentially, the apparatus 120 is configured to perform the method 100 of FIG. 1 A.Thus, the apparatus 120 may include components for carrying out the steps of the method 100 of FIG. 1A, that are applicable to the apparatus 120 even though some of the corresponding descriptions may be omitted here.

[0052] As seen in FIG. IB, the apparatus 120 includes a computer 122 configured to calculate, based on upper-limb virtual keypoint locations from 3D full-body virtual keypoint locations of the primate subject, a wrist joint center and an elbowjoint center (e g. as in Step 104 of FIG. 1A); refine the wrist joint center based on the elbow joint center, a median forearm length, and a directional vector, to obtain a refined wrist joint center (e g. as in Step 106 of FIG. 1A); determine a rotated unit vector based on MCP joint positions from the 3D full-body virtual keypoint locations of the primate subject, the directional vector and a forearm cone (e.g. as in Step 108 of FIG. 1A); and determine refined specific virtual keypoint locations from the upper-limb virtual keypoint locations based on the refined wrist joint center, the rotated unit vector and a median wrist radius (e.g. as in Step 110 of FIG. 1A). The MCP joint positions are for marking fingers of the primate subject. The directional vector is associated with the elbow joint center and the wrist joint center. The forearm cone is formable by the elbow joint center, the refined wrist joint center, and a selection of the upper-limb virtual keypoint locations.

[0053] The selection of the upper-limb virtual keypoint locations may include specific relevant upper-limb virtual keypoint locations.

[0054] The computer 122 may further be configured to derive the directional vector from the elbow joint center pointing to the wrist joint center; calculate a distance between the elbow joint center and the wrist joint center to obtain a forearm length; normalize the directional vector to obtain a unit vector; and offset the elbow joint center with the unit vector scaled by the median forearm length along a direction of the unit vector. The median forearm length may be determined from a plurality of forearm lengths. The plurality of forearm lengths may include the forearm length being the distance between the elbow joint center and the wrist joint center, and one or more other forearm lengths calculated from one or more recorded trajectories of full-body markerless tracking of the primate subject. Each of the one or more other forearm lengths may be a distance between an elbow joint center and a wrist joint center, both derived from each of the one or more recorded trajectories.

[0055] To calculate the wrist joint center, the computer 122 may be configured to determine a midpoint of the RSP and the USP. To calculate the elbow joint center, the computer 122 may be configured to determine a midpoint of the HLE and the HME.

[0056] To determine the rotated unit vector, the computer 122 may be configured to form the forearm cone including a tip defined by the elbow joint center and a circular base defined by the refined wrist joint center, the RSP and the USP; obtain a direction vector pointing from one of the MCP joint positions to a midpoint of a pair of other adjacent MCP joint positions; project the direction vector to a plane of the circular base of the forearm cone to obtain a projected vector; subsequently normalize the projected vector to obtain a normalized vector; and rotate the normalized vector with a constant amount of offset angle around a core axis of the forearm cone in a pronation or supination direction to obtain the rotated unit vector. The circular base and the median wrist radius may be as defined above for the method 100 of FIG. 1A.

[0057] To determine the refined specific virtual keypoint locations, the computer 122 may be configured to offset the refined wrist joint center with the rotated unit vector scaled by the median wrist radius along a direction of the rotated unit vector to obtain a refined ulnarstyloid process marker (USPnew) and along an opposite direction of the rotated unit vector to obtain a refined radius styloid process marker (RSPnew).

[0058] In various embodiments, the apparatus 120 may further include a plurality of colour video cameras 124 configured to capture the primate subject as series of 2D images. The plurality of colour video cameras 124 may be coupled to the computer 122 as denoted by line 128. Depending on the setup of the plurality of colour video cameras 124, some of these video cameras 124 may be suitably positioned for hand tracking, while others may be suitably positioned for full-body tracking. The computer 122 may be further configured to receive the sequences of 2D images captured by the plurality of colour video cameras 124; for each 2D image captured by each colour video camera 124, generate, using a trained machine learning model, a 2D bounding box; for each 2D image, generate, by the trained machine learning model, a plurality of heatmaps with scores of confidence; for each heatmap, select a pixel with the highest score of confidence, and associate the selected pixel to the keypoint to determine the 2D location of the keypoint; and based on the series of 2D images captured by the plurality of colour video cameras 124, triangulate the respective determined 2D locations to predict a sequence of 3D locations of the keypoint to infer the 3D full-body virtual keypoint locations.

[0059] The trained machine learning model is trained using at least a training dataset generated by the computer further configured to obtain at least one augmented dataset comprising projected-marker-based annotated images of a plurality of test subjects, and sample from the at least one augmented dataset and at least one in-the-wild dataset based on a sampling ratio to generate the training dataset. Each heatmap, the scores of confidence, the at least one in-the-wild dataset, the projected-marker-based annotated images of the plurality of test subjects, and the manually annotated images of the random subjects may be as defined above for the method 100 of FIG. 1 A. One or more of the augmented keypoints may respectively coincide with one or more of the 2D manually -annotated keypoint locations.

[0060] In various embodiments, the apparatus 120 may further include an optical markerbased motion capture system 126 configured to capture a plurality of physical markers over a period of time. Each physical marker may be placed on a bone landmark or a keypoint of each of the plurality of test subjects, and may be captured as a captured 3D trajectory. Theoptical marker-based motion capture system 126 may be coupled to the computer 122 as denoted by line 130. The optical marker-based motion capture system 126 may be configured to work in cooperation with plurality of colour video cameras 124 to infer the 3D full-body virtual keypoint locations. The plurality of colour video cameras 124 may further be configured to capture the plurality of test subjects over the period of time as sequences of 2D images.

[0061] To obtain the at least one augmented dataset, the computer 122 may further be configured to receive the sequences of 2D images captured by the plurality of colour video cameras 124 and the respective 3D trajectories captured by the optical marker-based motion capture system 126; for each physical marker, identify the captured 3D trajectory with a marker label representative of the bone landmark or keypoint on which the physical marker may be placed; obtain augmented markers, each augmented marker being calculated from positions of two or more of the physical markers placed on each test subject; determine an augmented 3D trajectory with an augmented label representative of each augmented marker; for each physical marker, project the captured 3D trajectory and for each augmented marker, project the augmented 3D trajectory to each of the 2D images to determine a 2D location in each 2D image; for each physical marker and for each augmented marker, based on the respective 2D locations in the sequences of 2D images and an exposure-related time of the plurality of colour video cameras, interpolate a 3D position for each of the 2D images, for each 2D image, based on the respective interpolated 3D positions of the plurality of physical markers and the augmented markers, and an extended volume, generate each projected-marker-based bounding box around each test subject; and generate the augmented dataset.

[0062] The augmented dataset may include at least one 2D image selected from the sequences of 2D images, the determined 2D location of each physical marker in the selected at least one 2D image, the determined 2D location of each augmented marker in the selected at least one 2D image, and the projected-marker-based bounding boxes for the selected at least one 2D image. The extended volume may be derived from two or more of the physical markers and / or the augmented markers having an anatomical, functional and / or structural relationship with one another. For each physical marker, the marker label may be arranged to be propagated with each determined 2D location such that in theaugmented dataset, each determined 2D location of each physical marker may contain the corresponding marker label. For each augmented marker, the augmented label may be arranged to be propagated with each determined 2D location such that in the augmented dataset, each determined 2D location of each augmented marker may contain the corresponding augmented label. The 2D marker-based keypoint locations may include the determined 2D locations of the plurality of physical markers. The augmented keypoints may include the determined 2D locations of the augmented markers.

[0063] To triangulate the respective determined 2D locations, the computer 122 may be configured to perform strategic triangulation of one trajectory of the keypoint at a time.

[0064] FIG. 2A shows a flow chart illustrating a method 200 for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of a primate subject, according to various embodiments. As seen in FIG. 2A, at Step 202, refined specific virtual keypoint locations are determined using a method 100 of FIG. 1A. The refined specific virtual keypoint locations include a refined ulnar styloid process marker (USPnew) and a refined radius styloid process marker (RSPnew). At Step 204, an orientation of a hand segment is calculated based on the RSPnew, the refined wrist joint center, and a half metacarpophalangeal marker (HMC) to determine an x-axis, a y-axis and a z-axis of the hand segment. At Step 206, an orientation of a forearm segment is calculated based on the RSPnew, the refined wrist joint center, and the elbow joint center to determine an x-axis, a y-axis and a z-axis of the forearm segment. At Step 208, 90 degrees is subtracted from a measurement angle between one of the x-axis, y-axis or z-axis of the hand segment and one of the x-axis, y-axis or z-axis of the forearm segment to enhance measurement of one of the distal upper-limb joint angles (e.g. see Equations (7) and (8) that will be described later below). The measurement angle forms an anatomically meaningful relationship between the hand segment and the forearm segment.

[0065] It should be appreciated that the 90-degree subtraction is used to allow the most normal / relax posture produce the wrist angle value near 0 degree. For example, if the 0-degree angle is defined to start at a different angle, it is possible to use a different number to add / subtract (for relative offsetting) or even, use an entirely different formulation for calculation. However, it is noted that when any axis from hand segment and any axis from forearm segment are chosen, and the angle between the two are calculated, some axis pairdo really make a meaningful angle. The selection of a value to subtract is not applicable only when the selection of axis pair does not make sense. When an axis pair makes sense, it is possible to find a subtraction value that causes the resting / relaxing posture to give a final output (wrist angle value) very close to 0 degree.

[0066] In the context of various embodiments, the term “meaningful angle” may refer to an angle within a range of angles between two meaningful axes that support one-to-one mapping from the entire range of the target range of motion task of that joint. For example, for a target angle in mind such as wrist flexion / extension, if a meaningful axis pair is picked, the measured angle between this meaningful axis pair cannot be repeated and must move in one way only when the wrist joint moves from fully flexed all the way to fully extended. On the other hand, if a bad (or non-meaningful) axis pair is picked, the measured angle may stay at just one number, or start at a number that runs to zero and ramps up again.

[0067] Orientation of the tracked hand inherently contains some information on forearm orientation that may be used to constrain and rectify forearm or wrist markers. By integrating hand-tracking information into the motion capture process, the fusion technique described by the method 200 excels in capturing movements related to forearm pronation / supi nation and wrist joint angles. More accurate wrist markers advantageously lead to more accurate calculations of forearm pronation / supination and wrist joint angles. In other words, the method 200 uses hand-tracking data to improve the accuracy of upper limb tracking, and overcomes the limitations of existing sensor-less motion capture systems in accurately detecting movement along the axial rotation of the forearm, a capability essential for comprehensive upper-limb tracking in medical contexts. The results obtained by using the method 200 are comparable to a marker-based motion capture system without the need for obtrusive wearable sensors or markers attached to the primate subject’s body.

[0068] In various embodiments, subtracting 90 degrees from the measurement angle at Step 208 may include subtracting 90 degrees from a first angle between the y-axis of the hand segment and the z-axis of the forearm segment to obtain a distal upper-limb joint angle being an angle of wrist flexion / extension; and / or subtracting 90 degrees from a second angle between the x-axis of the hand segment and the z-axis of the forearm segmentto obtain another distal upper-limb joint angle being an angle of wrist radial / ulnar deviation.

[0069] Calculating the orientation of the hand segment at Step 204 may include deriving the z-axis of the hand segment pointing from the HMC to the refined wrist joint center; deriving the y-axis of the hand segment in a palmar direction of a cross product between the z-axis of the hand segment and a direction from the refined wrist joint center to RSPnew; and deriving the x-axis of the hand segment pointing in a direction of a cross product between the y-axis of the hand segment and the z-axis of the hand segment.

[0070] Calculating the orientation of the forearm segment at Step 206 may include deriving the z-axis of the forearm segment pointing from the refined wrist joint center to the elbow joint center; deriving the y-axis of the forearm segment in a direction of a cross product between the z-axis of the forearm segment and a direction from the RSPnewto the refined wrist joint center, and deriving the x-axis of the forearm segment pointing in a direction of a cross product between the y-axis of the forearm segment and the z-axis of the forearm segment.

[0071] In various embodiments, the HMC may include a HMC2 estimated by shifting a midpoint of a metacarpophalangeal joint position (MCP2) marking an index finger on a hand of the primate subject and a metacarpophalangeal joint position (MCP3) marking a middle finger of the hand by a constant distance. The constant distance may be determined by an average thickness of the hand and a radius of the HMC2. In other embodiments, the HMC may be an alternative or nearby HMC that is anatomically similar to the HMC2. This may occur when or if certain metacarpophalangeal joint position(s) of the primate subject may be absent or not suitable.

[0072] The method 200 may further include determining changes in a relative orientation of the forearm segment with respect to an orientation of an upper arm segment of the primate subject to form a relative rotation matrix between the forearm segment and the upper arm segment; and extracting Euler angles from the relative rotation matrix to obtain the body segment rolling angles of the primate subject.

[0073] The orientation of upper arm segment may be defined as follows. First, with reference to FIG. 3 depicting a schematic representation of marker locations of a primatesubject, the calculation of the shoulder joint center (for each of the left side and the right side) may involve the following formulation:• obtaining the median distance between LACR and RACR from multiple frames (one or more records) from the primate subject;• calculating the offset direction from position of 4 torso markers, d = normalize((T10+XPRO)-(C7+STER));• determining right shoulder joint center = RACR + d*(0.17*(median ACR distance)+markerRadius);• determining left shoulder joint center = LACR + d*(0.17*(median ACR distance)+markerRadius).Second, a forearm pointing direction (for each side) is calculated by normalization of (wrist joint center - elbow joint center). Third, an upper arm pointing direction (for each side) is calculated by normalization of (elbow joint center- wrist joint center). Fourth, Z axis of upper arm is defined by the direction that points from elbow joint center to shoulder joint center. Fifth, the elbow bending angle is determined by calculating the angle between the forearm pointing direction and the upper arm pointing direction of the same side, and the following conditional approach is carried out to define X axis and Y axis of the upper arm.

[0074] If the elbow bends with sufficient degrees, e g. more than 20 degrees, the X-axis of the upper arm is the cross product between the forearm pointing direction and the Z axis of the upper arm, while the Y-axis of the upper arm is the cross product between the Z axis of the upper arm and the X axis of the upper arm. On the other hand, if the elbow does not bend more than 20 degrees, HME, HLE, and shoulder joint center are used instead where Y-axis of the right upper arm is pointing in the direction of the cross product between the Z-axis of the upper arm and the vector (HLE - elbow joint center), Y-axis of the left upper arm is pointing in the direction of the cross product between the vector (HLE - elbow joint center) and the Z-axis of the upper arm, while X-axis of the upper arm (for each side) is the cross product between the Y axis of the upper arm and the Z axis of the upper arm. With the X, Y, and Z axes of the upper arm, the orientation of the upper arm is established.

[0075] The body segment rolling angles may include an angle of forearm

[0076] FIG. 2B shows a schematic view of a system 240 for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of a primate subject, according to various embodiments.

[0077] Essentially, the system 240 is configured to perform the method 200 of FIG 2A. Thus, the system 240 may include components for carrying out the steps of the method 200 of FIG. 2A, that are applicable to the system 240 even though some of the corresponding descriptions may be omitted here.

[0078] As seen in FIG. 2B, the system 240 includes an apparatus 120 of FIG. IB configured to determine refined specific virtual keypoint locations including a refined ulnar styloid process marker (USPnew) and a refined radius styloid process marker (RSPnew). The computer 122 is further configured to calculate an orientation of a hand segment to determine an x-axis, a y-axis and a z-axis of the hand segment based on the RSPnew, the refined wrist joint center, and a HMC (e.g. Step 204 of FIG. 2 A); calculate an orientation of a forearm segment to determine an x-axis, a y-axis and a z-axis of the forearm segment based on the RSPnew, the refined wrist joint center, and the elbow joint center (e.g. Step 206 of FIG. 2A); and subtract 90 degrees from a measurement angle between one of the x-axis, y-axis or z-axis of the hand segment and one of the x-axis, y-axis or z-axis of the forearm segment to enhance measurement of one of the distal upper-limb joint angles (e g. Step 208 of FIG. 2A). As defined earlier, the measurement angle forms an anatomically meaningful relationship between the hand segment and the forearm segment. The computer 122 is configured to work cooperatively with the plurality of colour video cameras 124 to perform the steps of refining specific virtual keypoint locations, as defined by the method 100 of FIG. 1A.

[0079] To subtract 90 degrees from the measurement angle, the computer 122 may be configured to subtract 90 degrees from a first angle between the y-axis of the hand segment and the z-axis of the forearm segment to obtain a distal upper-limb joint angle being an angle of wrist flexion / extension; and / or subtract 90 degrees from a second angle between the x-axis of the hand segment and the z-axis of the forearm segment to obtain another distal upper-limb joint angle being an angle of wrist radial / ulnar deviation.

[0080] To calculate the orientation of the hand segment, the computer 122 may be configured to derive the z-axis of the hand segment pointing from the HMC to the refinedwrist joint center; derive the y-axis of the hand segment in a palmar direction of a cross product between the z-axis of the hand segment and a direction from the refined wrist joint center to RSPnewand derive the x-axis of the hand segment pointing in a direction of a cross product between the y-axis of the hand segment and the z-axis of the hand segment.

[0081] To calculate the orientation of the forearm segment, the computer 122 may be configured to derive the z-axis of the forearm segment pointing from the refined wrist joint center to the elbow joint center; derive the y-axis of the forearm segment in a direction of a cross product between the z-axis of the forearm segment and a direction from the RSPnewto the refined wrist joint center; and derive the x-axis of the forearm segment pointing in a direction of a cross product between the y-axis of the forearm segment and the z-axis of the forearm segment.

[0082] To estimate the HMC including a HMC2, the computer 122 may be configured to shift a midpoint of MCP2 marking an index finger on a hand of the primate subject and MCP3 marking a middle finger of the hand by a constant distance. The constant distance may be determined by an average thickness of the hand and a radius of the HMC2. In a different embodiment, the HMC may include an alternative or nearby HMC that may be anatomically similar to the HMC2.

[0083] In relation to the system 240, the computer may further be configured to determine changes in a relative orientation of the forearm segment with respect to an orientation of an upper arm segment of the primate subject to form a relative rotation matrix between the forearm segment and the upper arm segment; and extract Euler angles from the relative rotation matrix to obtain the body segment rolling angles of the primate subject. As defined earlier, the body segment rolling angles may include an angle of forearm pronation / supination.

[0084] While the methods 100, 200 described above are illustrated and described as a series of steps or events, it will be appreciated that any ordering of such steps or events are not to be interpreted in a limiting sense. For example, some steps may occur in different orders and / or concurrently with other steps or events apart from those illustrated and / or described herein. In addition, not all illustrated steps may be required to implement one or more aspects or embodiments described herein. Also, one or more of the steps depicted herein may be carried out in one or more separate acts and / or phases

[0085] Various embodiments further provide a computer program adapted to perform the methods 100, 200 (FIGS. 1A and 2A). A non-transitory computer readable medium including instructions which, when executed on a computer, cause the computer to perform the methods 100, 200 may also be provided. A data processing apparatus including means for carrying out the methods 100, 200 may be provided as well.

[0086] An example of a fusion of full-body and hand markerless mocap to enhance upperlimb track will be presented below. In this example, multiple synchronized and calibrated RGB cameras are given together with 3D trajectories of full-body virtual markers of a person (primate subject) being tracked by a markerless motion capture system (such as the one(s) described in International Application Nos. PCT / SG2025 / 050320 and / or PCT / SG2022 / 050398). This example is described in similar context to the methods 100, 200 of FIGS. 1 A and 2A and the apparatus 120 / system 240 of FIGS. IB and 2B, and thus, some corresponding descriptions may not be repeated here. The goal is to produce the forearm pronation / supination, and wrist joint angles that are closer to the values calculated from marker trajectories of marker-based motion capture system.

[0087] The contactless measurement approach of the fusion technique provides several advantages. Firstly, the recorded movement is not affected by the presence of wearable sensors or markers, as the sensors or markers may be accidentally shifted or dropped, requiring the subject to be extra careful with their movements. Secondly, there is no need for sensors or markers, significantly reducing the subject preparation time before each mocap session. Thirdly, being contactless makes it more hygienic, especially important in healthcare, as the mocap equipment does not need to be disinfected after every use, thereby reducing operation costs and time.

[0088] Compared to goniometer-based measurement, this fusion technique, according to various embodiments, is more effective This is because when using a goniometer, the subject must maintain a specific posture so that the goniometer is aligned with the relevant body parts at both ends of the range of motion. However, if the subject cannot maintain the proper posture, alignment becomes more difficult and less accurate. On the other hand, the fusion technique enables precise measurement of forearm pronation / supination during dynamic movement, without the limitations of the goniometer. Furthermore, the potential for human error in reading the measurements is eliminated3D Hand Joint Tracking

[0089] In the example, a pre-trained hand tracking model (such as Google’s MediaPipe) on each video record is applied and triangulation is performed to reconstruct the 3D trajectory of every joint of the target hand.Refine Wrist Markers

[0090] Reference is made to FIG. 3 which shows a schematic representation 301 illustrating the front and back skeleton views with marker locations produced by the fullbody markerless motion capture system, according to one example. To refine the wrist markers (RSP and USP), a total of seven 3D joint positions from body tracking (HLE, HME, RSP, USP) and hand tracking (MCP2, MCP3, MCP4) are used.

[0091] First, with reference to Step 104 of FIG. 1A, the location of the wrist joint center (WJC) is determined by finding the midpoint of RSP and USP, as given by Equation (1):Equation (1).

[0092] Similarly, the elbow joint center (EJC) is determined by finding the midpoint of HLE and HME, as given by Equation (2):Equation (2).

[0093] The forearm length ( / ) is then calculated as the distance between the EJC and WJC. To ensure robustness, with reference to Step 106 of FIG. 1A the median forearm length is used and the refined WJC is calculated as given by Equation (3):Equation (3), where V is the directional vector that points from EJC to WJCoriginal, given by Equation (4):v = WJCoriginal- EJCEquation (4).

[0094] A forearm cone is defined by EJC as the tip. The circular base of the cone is defined by the WJCrefined, RSP, and USP, with a radius calculated as half the distance between RSP and USP. To ensure the robustness of the computed radius, the median radius value (r) is used

[0095] Next, to find a directional vector that lies along the finger’s MCP joints, the MCP2, MCP3, and MCP4 joints from the hand tracking are used. It is noted that MCP5 is not used as the little finger tends to curl inwards towards the center of the palm when it flexes. Furthermore, the midpoint of MCP3 and MCP4 is chosen as it is more robust to noisy detection in the MCP3 and MCP4 joints. The directional vector that points from MCP2 to the midpoint of MCP 3 and MCP4 is projected to the plane of circular base of the cone and is normalized to a unit vector. This projected directional unit vector is rotated with a constant amount of offset angle around the core axis of the cone in the pronation direction. This constant offset angle is obtained from a set of healthy subjects. With reference to Steps 108, 110 of FIG. 1A, this rotated unit vector R is then used to calculate the new positions of the two wrist markers, as given in Equations (5) and (6):USPnew= WJCrefined+ r$\vec{R}$Equation (5), RSPnew= WJCrefined- r$\vec{R}$Equation (6).Estimate HMC2 Marker Location

[0096] The HMC2 marker’s position may be calculated more accurately through markerless hand tracking than through full-body markerless tracking. Thus, the available hand joint is used to estimate the location of the HMC2 marker.

[0097] To estimate the HMC2 marker’s location, the midpoint of MCP2 and MCP3 is shifted by a constant distance in the dorsal direction. The constant distance is determined by the average thickness of the human hand and the radius of the marker used. Essentially, the HMC2 is a refined position.Joint Angle Calculation

[0098] With the refined wrist markers (USPnew and RSPnew), the angle of forearm pronation / supination may be calculated by using the relative rotation matrix between the upper arm and forearm segments and then extracting the Euler angles from the relative rotation matrix.

[0099] With reference to Step 208 of FIG. 2A, to measure the angle of wrist flexion / extension, the angle between the y-axis of the hand segment (points in the palmar direction) and the z-axis of the forearm segment (points from WJC to EJC) is subtracted by 90 degrees, as given by Equation (7):wristflexion / extension= arccos( handy· forearmz / |handy||forearmz| ) - π / 2|handy||forearmz|Equation (7).

[0100] To measure the angle of wrist radial / ulnar deviation, the angle between the x-axis of the hand segment and the z-axis of the forearm segment is also subtracted by 90 degrees, as given by Equation (8):Equation (8).

[0101] For this example, with reference to Step 204 of FIG. 2A, the orientation of the hand segment is calculated using RSPnew, WJCrefined, and HMC2. For the right hand, the z-axis points from HMC2 to WJCrefined, the y-axis points in the palmar direction (calculated from a cross product between the z-axis and the direction from WJCrefinedto RSPnew), and the x- axis points in the direction of the cross product between y-axis and z-axis.

[0102] With reference to Step 206 of FIG. 2A, the orientation of the forearm segment is calculated using RSPnew, WJCrefined, and EJC. For the right forearm, the z-axis points from WJCrefinedto EJC, the y-axis points in the direction of the cross product between the z-axis and the direction from RSPnew to WJCrefined, and the x-axis points in the direction of the cross product between y-axis and z-axisEvaluation Experiment

[0103] An experiment was conducted to quantify the effectiveness of the proposed fusion method. The aim is to compare the joint angle values obtained from three different sources. The first source is the direct calculation from a marker-based motion capture system (Qualisys), which served as the ground truth in this experiment. The second source is the direct calculation from the virtual markers retrieved from a full-body markerless motion capture system without any marker refinement. The third source uses the proposed fusion technique, which integrates hand tracking for some marker adjustments before using the same set of formulas to calculate the joint angle.Camera Setup

[0104] The markerless system makes use of eight (8) RGB cameras (e.g. the plurality of colour video cameras 124 of FIGS. IB and 2B) that are synchronized and calibrated. Four of these cameras are placed at close proximity (about 0.9 to 1.5 meters from the hand) to capture clear images of the hand around the center of the field of view. To ensure simultaneous recording with the marker-based motion capture system (e.g. the optical marker-based motion capture system 126 of FIG. IB), all the RGB cameras are synchronized at the frame level, running at 50 Hz, while the marker-based motion capture system runs at 200 Hz.Movement Set

[0105] Five subjects were recruited for this experiment. Each subject performed 2 sets of tasks. The first set was active range of motion (AROM) tasks where the subject performed each of the following items 3 times to both extremes:• wrist flexion / extension• wrist radial / ulnar deviation• forearm pronation / supination.

[0106] The second set was the upper limb functional tasks. The subject performed each of the following items 3 times:• hand to head (simulate grooming)• hand to nose (simulate feeding)• hand to contralateral shoulder (simulate cleaning of the body)• hand to waist (simulate putting on pants after toileting).Marker Set

[0107] A total of 20 markers were used in this experiment. The position of these markers are seen in FIG. 3, as follows:• Throax: STER, XPRO, C7, T4, T8, T10• Shoulder: RACR, LACR• Elbow: RHLE, RHME, LHLE, LHME• Wrist: RRSP, RUSP, LRSP, LUSP• Hand: RCAP, LCAP, RHMC2, LHMC2Results and Discussion

[0108] The effect of without and with hand tracking methods shows clear differences as depicted in FIGS. 4A and 4B, as well as in the joint angles plots of FIGS. 5 to 7 (presenting the comparison of joint angle sequences in the range of motion (ROM) tasks from three different methods).

[0109] FIG. 4A shows a tracking display 401 (left) of the right forearm 401 ’ of a primate subject (right), where the right forearm 401’ is in a neutral position, according to an example. When the right forearm 401’ is in the neutral position, the RRSP markers from both the full-body tracking and the hand-tracking are close to each other (see the white arrows in the tracking display 401).

[0110] FIG. 4B shows a tracking display 403 (left) of the right forearm 403’ of the primate subject (right) of FIG. 4 A, where the right forearm 403’ is pronated, according to an example When the right forearm 403’ is pronated, the RRSP marker from the full -body tracking is incorrect as it remains at the neutral position (e g in FIG. 4A), but this may be corrected by considering the results of hand tracking (see the white arrows in the tracking display 403).

[0111] FIG. 5 shows a graph 501 illustrating plots comparing joint angle sequences for wrist extension / flexion using three different methods, namely the ground truth 503, without hand tracking 505, and with hand tracking 507, according to an example.

[0112] FIG. 6 shows a graph 601 illustrating plots comparing joint angle sequences for wrist radial / ulnar deviation using the three different methods, namely the ground truth 603, without hand tracking 605, and with hand tracking 607, according to an example.

[0113] FIG. 7 shows a graph 701 illustrating plots comparing joint angle sequences for forearm supination / pronation using the three different methods, namely the ground truth 703, without hand tracking 705, and with hand tracking 707, according to an example.

[0114] The method that combined full-body markerless data with markerless hand tracking 507, 607, 707 produces joint angles with less noise. The joint angles from this method 507, 607, 707 are closer to the joint angles calculated from marker-based motion capture (ground truth 503, 603, 703). The fusion method 707 is especially effective in tracking forearm pronation / supination, which was previously impossible using full-body tracking alone 705. The highlighted sections on pronation / supination plots the graph 701 of FIG. 7 are when the USP marker drifts away from its bone landmark causing the ground truth calculation to not reach more supination as it should. Therefore, these sections are removed from the benchmarking analysis.

[0115] FIGS. 8A and 8B show scatter plots of markerless measurement, without hand tracking fusion 801 and with hand tracking fusion 803, respectively, against marker-based measurement (ground truth) for wrist flexion / extension angle. The correlation coefficients of the scatter plots of markerless measurement without hand tracking fusion 801 and with hand tracking fusion 803 are about 0.899 and 0.974, respectively. It is observed that the fusion method 803 enhances the measurement correlation against the ground truth for wrist flexion / extension.

[0116] FIGS. 9A and 9B show scatter plots of markerless measurement, without hand tracking fusion 901 and with hand tracking fusion 903, respectively, against marker-based measurement (ground truth) for wrist radial / ulnar deviation angle. The correlation coefficients of the scatter plots of markerless measurement without hand tracking fusion 901 and with hand tracking fusion 903 are about 0.868 and 0.912, respectively. It is observed that the fusion method 903 enhances the measurement correlation against the ground truth for wrist radial / ulnar deviation.

[0117] FIGS. 10A and 10B show scatter plots of markerless measurement, without hand tracking fusion 1001 and with hand tracking fusion 1003, respectively, against marker-based measurement (ground truth) for forearm pronation / supination angle. Neutral angle is 90 degrees. Any point with ground truth below 45 degree are removed from this analysis because the USP marker always drift away from its bone landmark when the surrounding skin is pulled by the supination action causing the ground truth to be unreliable. The correlation coefficients of the scatter plots of markerless measurement without hand tracking fusion 1001 and with hand tracking fusion 1003 are about 0.169 and 0.947, respectively. It is observed that the fusion method 1003 significantly enhances the measurement correlation against the ground truth for forearm pronation / supination angle. This finding is in line with the graph 701 of FIG. 7 illustrating plots comparing the joint angle sequences for forearm supination / pronation, and the tracking display 403 of FIG. 4B of the pronated right forearm 403’ of the primate subject.Correlation with Marker-based Mocap

[0118] The proposed fusion method has demonstrated a marked improvement in correlation for all range of motion (ROM) tasks. When hand tracking is used, all correlation coefficient values exceed 0.9, with a p-value of less than 0.001 (see FIGS. 8B, 9B, and 10B). There is a significant improvement in the correlation coefficient value for forearm pronation / supination, which is 0.947 with hand tracking compared to 0.169 without hand tracking (see FIGS. 10A and 10B).Joint Angle Accuracy

[0119] Tables 1 and 2 respectively present the results for range of motion (ROM) and activities of daily living (ADL) tasks.

[0120] Table 1: Comparison of joint angle root mean square error (RMSE) during ROM tasks

[0121] Table 2: Comparison of joint angle root mean square error (RMSE) during ADL tasks

[0122] These results quantify the deviation of joint angles from the ground truth values calculated from marker-based motion capture, using the root mean square error (RMSE) as the measurement metric.Potential Extension

[0123] With large enough database of MCP2, MCP3, MCP4, and MCP53D joint positions together with the ground truth RSP and USP markers from a larger population, a machine learning method may be used to produce an accurate mapping function that provide the right offset angle for the most accurate pronation / supination.

[0124] The fusion method may be applied in a large capture volume with the help of image super-resolution techniques such as Nvidia’s Deep Learning Super Sampling (DLSS) or Al-based image restoration or super resolution technique. These techniques may enhance the resolution for hand tracking when the hand is too far from the cameras.Commercial Application

[0125] One commercial application of the fusion method is in the field of recovery assessment and tracking. This approach enables healthcare professionals and researchers to gather detailed and accurate data on an individual’s recovery progress after an injury or medical intervention. The fusion of full-body and hand markerless motion capture allows for the precise measurement of various upper-limb movements and tasks, which provide valuable insights into the subject’s rehabilitation journey.

[0126] While the invention has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

CLAIMS1. A method of refining specific virtual keypoint locations of an upper limb of a primate subject for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of the primate subject, the method comprising:determining a median forearm length of the primate subject, a median wrist radius, and 3-dimensional (3D) full-body virtual keypoint locations of the primate subject comprising upper-limb virtual keypoint locations and metacarpophalangeal (MCP) joint positions marking fingers of the primate subject;calculating, based on the upper-limb virtual keypoint locations, a wrist joint center and an elbow joint center;refining the wrist joint center based on the elbow joint center, the median forearm length, and a directional vector associated with the elbow joint center and the wrist joint center, to obtain a refined wrist joint center;determining a rotated unit vector,wherein the rotated unit vector is associated with the MCP joint positions, the directional vector and a forearm cone, andwherein the forearm cone is formed by the elbow joint center, the refined wrist joint center, and a selection of the upper-limb virtual keypoint locations; andbased on the refined wrist joint center, the rotated unit vector, and the median wrist radius, determining refined specific virtual keypoint locations from the upper-limb virtual keypoint locations.

2. The method as claimed in claim 1,wherein the directional vector is pointing from the elbow joint center to the wrist joint center;wherein the median forearm length is determined from a plurality of forearm lengths, the plurality of forearm lengths comprising:a forearm length being a distance between the elbow joint center and the wrist joint center, andone or more other forearm lengths calculated from one or more recorded trajectories of full-body markerless tracking of the primate subject, each of the one or more other forearm lengths being a distance between an elbow joint center and a wrist joint center, both derived from each of the one or more recorded trajectories; andwherein the refined wrist joint center is calculated by normalizing the directional vector to obtain a unit vector, and offsetting the elbow joint center with the unit vector scaled by the median forearm length along a direction of the unit vector.

3. The method as claimed in claim 1 or 2,wherein the upper-limb virtual keypoint locations comprise an ulnar styloid process marker (USP), a radius styloid process marker (RSP), a medial epi-condyle marker (HME), and a lateral epicondyle marker (HLE), andwherein calculating the wrist joint center comprises determining a midpoint of the RSP and the USP, andcalculating the elbow joint center comprises determining a midpoint of the HLE and the HME.

4. The method as claimed in claim 3, further comprising forming the forearm cone comprising:a tip defined by the elbow joint center; anda circular base defined by the refined wrist joint center, the RSP and the USP, wherein the circular base has a radius being half a distance between the RSP and the USP, andwherein the rotated unit vector is determined by:obtaining a direction vector pointing from one of the MCP joint positions to a midpoint of a pair of other adjacent MCP joint positions,projecting the direction vector to a plane of the circular base of the forearm cone to obtain a projected vector,subsequently normalizing the projected vector to obtain a normalized vector, androtating the normalized vector with a constant amount of offset angle around a core axis of the forearm cone in a pronation or supination direction to obtain the rotated unit vector.

5. The method as claimed in claim 4,wherein the median wrist radius is determined from a plurality of wrist radius values, the plurality of wrist radius values comprising:the radius being half the distance between the RSP and the USP, and one or more other wrist radius values calculated from one or more recorded trajectories of full-body markerless tracking of the primate subject, each of the one or more other wrist radius values being half a distance between a RSP and a USP, both derived from each of the one or more recorded trajectories.

6. The method as claimed in any one of claims 1 to 5, wherein determining the refined specific virtual keypoint locations comprises offsetting the refined wrist joint center with the rotated unit vector scaled by the median wrist radius along a direction of the rotated unit vector to obtain a refined ulnar styloid process marker (USPnew) and along an opposite direction of the rotated unit vector to obtain a refined radius styloid process marker (RSPnew).

7. The method as claimed in any one of claims 1 to 6, wherein the 3D full-body virtual keypoint locations are generated by an inference method, the inference method comprising:based on the primate subject captured by a plurality of colour video cameras as series of 2D images,for each 2D image captured by each colour video camera, generating, using a trained machine learning model, a 2D bounding box;for each 2D image, generating, by the trained machine learning model, a plurality of heatmaps with scores of confidence,wherein each heatmap is for 2D localization of a keypoint of thefor each heatmap, selecting a pixel with the highest score of confidence, and associating the selected pixel to the keypoint, thereby determining the 2D location of the keypoint, wherein for each heatmap, the scores of confidence are indicative of probability of having the associated keypoint in different 2D locations in the generated 2D bounding box; andbased on the series of 2D images captured by the plurality of colour video cameras, triangulating the respective determined 2D locations to predict a sequence of 3D locations of the keypoint, thereby inferring the 3D full-body virtual keypoint locations.

8. The method as claimed in claim 7, wherein the trained machine learning model is trained using at least a training dataset generated by a generation method, the generation method comprising:obtaining at least one augmented dataset comprising projected-marker-based annotated images of a plurality of test subjects; andsampling from the at least one augmented dataset and at least one in-the-wild dataset based on a sampling ratio to generate the training dataset,wherein the at least one in-the-wild dataset comprises manually annotated images of random subjects under unrestricted conditions, the projected-marker-based annotated images of the plurality of test subjects comprise 2D marker- based keypoint locations, augmented keypoints and projected-marker-based bounding boxes,the manually annotated images of the random subjects comprise 2D manually-annotated keypoint locations and manually-annotated bounding boxes, andone or more of the augmented keypoints respectively coincide with one or more of the 2D manually-annotated keypoint locations.

9. The method as claimed in claim 8, wherein obtaining the at least one augmented dataset comprises:based on a plurality of physical markers captured by an optical marker-based motion capture system, each as a captured 3D trajectory, wherein each physical marker isplaced on a bone landmark or a keypoint of each of the plurality of test subjects, and the plurality of test subjects substantially simultaneously captured by the plurality of colour video cameras over a period of time as sequences of 2D images,for each physical marker, identifying the captured 3D trajectory with a marker label representative of the bone landmark or keypoint on which the physical marker is placed, obtaining augmented markers, each augmented marker being calculated from positions of two or more of the physical markers placed on each test subject;determining an augmented 3D trajectory with an augmented label representative of each augmented marker;for each physical marker, projecting the captured 3D trajectory and for each augmented marker, projecting the augmented 3D trajectory to each of the 2D images to determine a 2D location in each 2D image;for each physical marker and for each augmented marker, based on the respective 2D locations in the sequences of 2D images and an exposure-related time of the plurality of colour video cameras, interpolating a 3D position for each of the 2D images;for each 2D image, based on the respective interpolated 3D positions of the plurality of physical markers and the augmented markers, and an extended volume, generating each projected-marker-based bounding box around each test subject, wherein the extended volume is derived from two or more of the physical markers and / or the augmented markers having an anatomical, functional and / or structural relationship with one another, andgenerating the augmented dataset comprising at least one 2D image selected from the sequences of 2D images, the determined 2D location of each physical marker in the selected at least one 2D image, the determined 2D location of each augmented marker in the selected at least one 2D image, and the projected-marker-based bounding boxes for the selected at least one 2D image,wherein for each physical marker, the marker label is arranged to be propagated with each determined 2D location such that in the augmented dataset, each determined 2D location of each physical marker contains the corresponding marker label,wherein for each augmented marker, the augmented label is arranged to be propagated with each determined 2D location such that in the augmented dataset, eachdetermined 2D location of each augmented marker contains the corresponding augmented label,wherein the 2D marker-based keypoint locations comprise the determined 2D locations of the plurality of physical markers, andwherein the augmented keypoints comprise the determined 2D locations of the augmented markers.

10. The method as claimed in any one of claims 7 to 9, wherein triangulating the respective determined 2D locations comprises performing strategic triangulation of one trajectory of the keypoint at a time.

11. A method for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of a primate subject, the method comprising:determining refined specific virtual keypoint locations using a method as claimed in any one of claims 1 to 10, wherein the refined specific virtual keypoint locations comprise a refined ulnar styloid process marker (USPnew) and a refined radius styloid process marker (RSPnew);based on the RSPnew, the refined wrist joint center, and a half metacarpophalangeal marker (HMC), calculating an orientation of a hand segment to determine an x-axis, a y-axis and a z-axis of the hand segment;based on the RSPnew, the refined wrist joint center, and the elbow joint center, calculating an orientation of a forearm segment to determine an x-axis, a y-axis and a z-axis of the forearm segment; andsubtracting 90 degrees from a measurement angle between one of the x-axis, y-axis or z-axis of the hand segment and one of the x-axis, y-axis or z-axis of the forearm segment to enhance measurement of one of the distal upper-limb joint angles, wherein the measurement angle forms an anatomically meaningful relationship between the hand segment and the forearm segment.

12. The method as claimed in claim 11, wherein subtracting 90 degrees from the measurement angle comprises:subtracting 90 degrees from a first angle between the y-axis of the hand segment and the z-axis of the forearm segment to obtain a distal upper-limb joint angle being an angle of wrist flexion / extension; and / orsubtracting 90 degrees from a second angle between the x-axis of the hand segment and the z-axis of the forearm segment to obtain another distal upper-limb joint angle being an angle of wrist radial / ulnar deviation.

13. The method as claimed in claim 11 or 12, wherein calculating the orientation of the hand segment comprises:deriving the z-axis of the hand segment pointing from the HMC to the refined wrist joint center;deriving the y-axis of the hand segment in a palmar direction of a cross product between the z-axis of the hand segment and a direction from the refined wrist joint center to RSPnew; andderiving the x-axis of the hand segment pointing in a direction of a cross product between the y-axis of the hand segment and the z-axis of the hand segment.14 The method as claimed in any one of claims 11 to 13, wherein calculating the orientation of the forearm segment comprises:deriving the z-axis of the forearm segment pointing from the refined wrist joint center to the elbow joint center;deriving the y-axis of the forearm segment in a direction of a cross product between the z-axis of the forearm segment and a direction from the RSPnewto the refined wrist joint center; andderiving the x-axis of the forearm segment pointing in a direction of a cross product between the y-axis of the forearm segment and the z-axis of the forearm segment.

15. The method as claimed in any one of claims 11 to 14, wherein the HMC comprises a HMC2 estimated by:shifting a midpoint of a metacarpophalangeal joint position (MCP2) marking an index finger on a hand of the primate subject and a metacarpophalangeal joint position (MCP3) marking a middle finger of the hand by a constant distance,wherein the constant distance is determined by an average thickness of the hand and a radius of the HMC2.

16. The method as claimed in claim 15, wherein the HMC comprises an alternative or nearby HMC that is anatomically similar to the HMC2.

17. The method as claimed in any one of claims 11 to 16, further comprising:determining changes in a relative orientation of the forearm segment with respect to an orientation of an upper arm segment of the primate subject to form a relative rotation matrix between the forearm segment and the upper arm segment; andextracting Euler angles from the relative rotation matrix to obtain the body segment rolling angles of the primate subject.

18. The method as claimed in claim 17, wherein the body segment rolling angles comprise an angle of forearm pronation / supination.

19. An apparatus for refining specific virtual keypoint locations of an upper limb of a primate subject for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of the primate subject, the apparatus comprising:a computer configured to:calculate, based on upper-limb virtual keypoint locations from 3- dimensional (3D) full-body virtual keypoint locations of the primate subject, a wrist joint center and an elbow joint center;refine the wrist joint center based on the elbow joint center, a median forearm length, and a directional vector, to obtain a refined wrist joint center; determine a rotated unit vector based on metacarpophalangeal (MCP) joint positions from the 3D full-body virtual keypoint locations of the primate subject, the directional vector and a forearm cone,wherein the MCP joint positions are for marking fingers of the primate subject,the directional vector is associated with the elbow joint center and the wrist joint center, andthe forearm cone is formable by the elbow joint center, the refined wrist joint center, and a selection of the upper-limb virtual keypoint locations; anddetermine refined specific virtual keypoint locations from the upper-limb virtual keypoint locations based on the refined wrist joint center, the rotated unit vector and a median wrist radius.

20. The apparatus as claimed in claim 19,wherein the computer is further configured to:derive the directional vector from the elbow joint center pointing to the wrist joint center;calculate a distance between the elbow joint center and the wrist joint center to obtain a forearm length;normalize the directional vector to obtain a unit vector; andoffset the elbow joint center with the unit vector scaled by the median forearm length along a direction of the unit vector, andwherein the median forearm length is determined from a plurality of forearm lengths, the plurality of forearm lengths comprising:the forearm length being the distance between the elbow joint center and the wrist joint center, andone or more other forearm lengths calculated from one or more recorded trajectories of full-body markerless tracking of the primate subject, each of the one or more other forearm lengths being a distance between an elbow joint center and a wrist joint center, both derived from each of the one or more recorded trajectories.

21. The apparatus as claimed in claim 19 or 20,wherein the upper-limb virtual keypoint locations comprise an ulnar styloid process marker (USP), a radius styloid process marker (RSP), a medial epi-condyle marker (HME), and a lateral epicondyle marker (HLE);wherein to calculate the wrist joint center, the computer is configured to determine a midpoint of the RSP and the USP, andwherein to calculate the elbow joint center, the computer is configured to determine a midpoint of the HLE and the HME.

22. The apparatus as claimed in claim 21,wherein to determine the rotated unit vector, the computer is configured to: form the forearm cone comprising a tip defined by the elbow joint center and a circular base defined by the refined wrist joint center, the RSP and the USP, wherein the circular base has a radius being half a distance between the RSP and the USP,obtain a direction vector pointing from one of the MCP joint positions to a midpoint of a pair of other adjacent MCP joint positions,project the direction vector to a plane of the circular base of the forearm cone to obtain a projected vector,subsequently normalize the projected vector to obtain a normalized vector, androtate the normalized vector with a constant amount of offset angle around a core axis of the forearm cone in a pronation or supination direction to obtain the rotated unit vector; andwherein the median wrist radius is determined from a plurality of wrist radius values, the plurality of wrist radius values comprising:the radius being half the distance between the RSP and the USP, and one or more other wrist radius values calculated from one or more recorded trajectories of full-body markerless tracking of the primate subject, each of the one or more other wrist radius values being half a distance between a RSP and a USP, both derived from each of the one or more recorded trajectories23. The apparatus as claimed in any one of claims 19 to 22, wherein to determine the refined specific virtual keypoint locations, the computer is configured to offset the refined wrist joint center with the rotated unit vector scaled by the median wrist radius along a direction of the rotated unit vector to obtain a refined ulnar styloid process marker (USPnew) and along an opposite direction of the rotated unit vector to obtain a refined radius styloid process marker (RSPnew).

24. The apparatus as claimed in any one of claims 19 to 23, further comprising a plurality of colour video cameras configured to capture the primate subject as series of 2D images, wherein the computer is further configured to:receive the sequences of 2D images captured by the plurality of colour video cameras;for each 2D image captured by each colour video camera, generate, using a trained machine learning model, a 2D bounding box;for each 2D image, generate, by the trained machine learning model, a plurality of heatmaps with scores of confidence,wherein each heatmap is for 2D localization of a keypoint of the primate subject;for each heatmap, select a pixel with the highest score of confidence, and associate the selected pixel to the keypoint to determine the 2D location of the keypoint, wherein for each heatmap, the scores of confidence are indicative of probability of having the associated keypoint in different 2D locations in the generated 2D bounding box; andbased on the series of 2D images captured by the plurality of colour video cameras, triangulate the respective determined 2D locations to predict a sequence of 3D locations of the keypoint to infer the 3D full-body virtual keypoint locations.

25. The apparatus as claimed in claim 24, wherein the trained machine learning model is trained using at least a training dataset generated by the computer further configured to:obtain at least one augmented dataset comprising projected- marker-based annotated images of a plurality of test subjects; andsample from the at least one augmented dataset and at least one in-the-wild dataset based on a sampling ratio to generate the training dataset,wherein the at least one in-the-wild dataset comprises manually annotated images of random subjects under unrestricted conditions, the projected-marker-based annotated images of the plurality of test subjects comprise 2D marker- based keypoint locations, augmented keypoints and projected-marker-based bounding boxes,the manually annotated images of the random subjects comprise 2D manually-annotated keypoint locations and manually-annotated bounding boxes, andone or more of the augmented keypoints respectively coincide with one or more of the 2D manually-annotated keypoint locations.

26. The apparatus as claimed in claim 25, further comprising an optical marker-based motion capture system configured to capture a plurality of physical markers over a period of time, wherein each physical marker is placed on a bone landmark or a keypoint of each of the plurality of test subjects, and is captured as a captured 3D trajectory; andthe plurality of colour video cameras is further configured to capture the plurality of test subjects over the period of time as sequences of 2D images,wherein to obtain the at least one augmented dataset, the computer is further configured to:receive the sequences of 2D images captured by the plurality of colour video cameras and the respective 3D trajectories captured by the optical marker-based motion capture system;for each physical marker, identify the captured 3D trajectory with a marker label representative of the bone landmark or keypoint on which the physical marker is placed;obtain augmented markers, each augmented marker being calculated from positions of two or more of the physical markers placed on each test subject;determine an augmented 3D trajectory with an augmented label representative of each augmented marker;for each physical marker, project the captured 3D trajectory and for each augmented marker, project the augmented 3D trajectory to each of the 2D images to determine a 2D location in each 2D image;for each physical marker and for each augmented marker, based on the respective 2D locations in the sequences of 2D images and an exposure-related time of the plurality of colour video cameras, interpolate a 3D position for each of the 2D images;for each 2D image, based on the respective interpolated 3D positions of the plurality of physical markers and the augmented markers, and an extended volume derived from two or more of the physical markers and / or the augmented markers having an anatomical, functional and / or structural relationship with one another, generate each project ed-marker-based bounding box around each test subject; andgenerate the augmented dataset comprising at least one 2D image selected from the sequences of 2D images, the determined 2D location of each physical marker in the selected at least one 2D image, the determined 2D location of each augmented marker in the selected at least one 2D image, and the projected-marker-based bounding boxes for the selected at least one 2D image,wherein for each physical marker, the marker label is arranged to be propagated with each determined 2D location such that in the augmented dataset, each determined 2D location of each physical marker contains the corresponding marker label,wherein for each augmented marker, the augmented label is arranged to be propagated with each determined 2D location such that in the augmented dataset, each determined 2D location of each augmented marker contains the corresponding augmented label,wherein the 2D marker-based keypoint locations comprise the determined 2D locations of the plurality of physical markers, andwherein the augmented keypoints comprise the determined 2D locations of the augmented markers.

27. The apparatus as claimed in any one of claims 24 to 26, wherein to triangulate the respective determined 2D locations, the computer is configured to perform strategic triangulation of one trajectory of the key point at a time.

28. A system for enhancing measurements of distal upper-limb joint angles and / or body segment rolling angles of a primate subject, the system comprising:an apparatus as claimed in any one of claims 19 to 27, wherein the apparatus is configured to determine refined specific virtual keypoint locations comprising a refined ulnar styloid process marker (USPnew) and a refined radius styloid process marker (RSPnew); andwherein the computer is further configured to:calculate an orientation of a hand segment to determine an x-axis, a y-axis and a z-axis of the hand segment based on the RSPnew, the refined wrist joint center, and a half metacarpophalangeal marker (HMC);calculate an orientation of a forearm segment to determine an x-axis, a y- axis and a z-axis of the forearm segment based on the RSPnew, the refined wrist joint center, and the elbow joint center; andsubtract 90 degrees from a measurement angle between one of the x-axis, y-axis or z-axis of the hand segment and one of the x-axis, y-axis or z-axis of the forearm segment to enhance measurement of one of the distal upper-limb joint angles, wherein the measurement angle forms an anatomically meaningful relationship between the hand segment and the forearm segment.

29. The system as claimed in claim 28, wherein to subtract 90 degrees from the measurement angle, the computer is configured to:subtract 90 degrees from a first angle between the y-axis of the hand segment and the z-axis of the forearm segment to obtain a distal upper-limb joint angle being an angle of wrist flexion / extension; and / orsubtract 90 degrees from a second angle between the x-axis of the hand segment and the z-axis of the forearm segment to obtain another distal upper-limb joint angle being an angle of wrist radial / ulnar deviation.

30. The system as claimed in claim 28 or 29, wherein to calculate the orientation of the hand segment, the computer is configured to:derive the z-axis of the hand segment pointing from the HMC to the refined wrist joint center;derive the y-axis of the hand segment in a palmar direction of a cross product between the z-axis of the hand segment and a direction from the refined wrist joint center to RSPnew; andderive the x-axis of the hand segment pointing in a direction of a cross product between the y-axis of the hand segment and the z-axis of the hand segment.

31. The system as claimed in any one of claims 28 to 30, wherein to calculate the orientation of the forearm segment, the computer is configured to:derive the z-axis of the forearm segment pointing from the refined wrist joint center to the elbow joint center;derive the y-axis of the forearm segment in a direction of a cross product between the z-axis of the forearm segment and a direction from the RSPnewto the refined wrist joint center; andderive the x-axis of the forearm segment pointing in a direction of a cross product between the y-axis of the forearm segment and the z-axis of the forearm segment.32 The system as claimed in any one of claims 28 to 31, wherein to estimate the HMC comprising a HMC2, the computer is configured to:shift a midpoint of a metacarpophalangeal joint position (MCP2) marking an index finger on a hand of the primate subject and a metacarpophalangeal joint position (MCP3) marking a middle finger of the hand by a constant distance,wherein the constant distance is determined by an average thickness of the hand and a radius of the HMC233. The system as claimed in claim 32, wherein the HMC comprises an alternative or nearby HMC that is anatomically similar to the HMC2.

34. The system as claimed in any one of claims 28 to 33, wherein the computer is further configured to:determine changes in a relative orientation of the forearm segment with respect to an orientation of an upper arm segment of the primate subject to form a relative rotation matrix between the forearm segment and the upper arm segment; andextract Euler angles from the relative rotation matrix to obtain the body segment rolling angles of the primate subject35. The system as claimed in claim 34, wherein the body segment rolling angles comprise an angle of forearm pronation / supination.

36. A computer program adapted to perform a method as claimed in any one of claims 1 to 18.

37. A non-transitory computer readable medium comprising instructions which, when executed on a computer, cause the computer to perform a method as claimed in any one of claims 1 to 18.

38. A data processing apparatus comprising means for carrying out a method as claimed in any one of claims 1 to 18.

Images