Mobility device, system for and method of person following

The autonomous mobility device uses LiDAR and camera modules for hybrid sensing to address the challenges of accurate person following in unstructured environments, enhancing safety and independence for individuals with motor impairments.

WO2026240701A1PCT designated stage Publication Date: 2026-11-19NANYANG TECH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/SG2026/050210
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-15
Filing Date
2026-03-31
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Conventional wheelchairs require mental and physical effort during movement, pose safety concerns due to lack of obstacle avoidance strategies, and struggle with accurate person following in unstructured environments due to false positives and sensor inaccuracies.

Method used

An autonomous mobility device equipped with a system that combines LiDAR and camera modules for hybrid sensing, using Extended Kalman Filters and multimodal sensor data fusion to track a target person, enabling robust person following by determining a target's position in 3D space and navigating accordingly.

Benefits of technology

The system effectively tracks and follows a target person in various environments, reducing physical strain on caregivers and enhancing safety and independence for individuals with motor impairments by allowing side-by-side monitoring and communication.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SG2026050210_19112026_PF_FP_ABST
    Figure SG2026050210_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A system and method of person following. The method comprises receiving at least one LiDAR detection data and at least one camera image data; based on the at least one LiDAR detection data, determining at least one first positional output corresponding to the target person; based on the at least one camera image data, determining at least one second positional output corresponding to the target person; using a tracker module, determining an estimated position of the target person relative to a mobility device based on the at least one first positional output and the at least one second positional output.
Need to check novelty before this filing date? Find Prior Art

Description

MOBILITY DEVICE, SYSTEM FOR AND METHOD OF PERSON FOLLOWINGCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority to the Singapore application no.10202501255S filed 15 May, 2025, the contents of which are hereby incorporated by reference in their entirety for all purposes.TECHNICAL FIELD

[0002] This application relates generally to the field of autonomous mobility device control, and more particularly, to an autonomous mobility device, a system for person following, and a method of person following.BACKGROUND

[0003] Conventional wheelchairs are often used in transporting a user with mobility issues (such as an elderly or an injured individual). However, conventional wheelchairs require mental and physical effort during movement and may cause physical strain to a caregiver or companion. Motorised wheelchairs pose safety concerns due to the lack of obstacle avoidance strategies and the need for environmental familiarity in order to navigate.

[0004] Autonomous mobility devices, such as autonomous wheelchairs, represent a transformative leap in assistive technology, offering elevated levels of independence and safety for individuals with severe motor impairments. In addition, autonomous mobility devices alleviate the need for the caregiver or companion to be positioned behind the wheelchair, which allows the caregiver or companion to walk alongside or nearby to the autonomous wheelchair user during movement. Further, automated person following may be performed by the autonomous wheelchair, which allows side-by-side monitoring and communication betweenthe caregiver and the user. However, various challenges are present in such a person following implementations, such as inaccuracies during tracking or person following due to false positives introduced by environmental artefacts.SUMMARY

[0005] According to an aspect, disclosed herein a system. The system comprises: memory storing instructions; and a processor coupled to the memory and configured to process the stored instructions to implement: a module configured to perform a method of person following. The method comprises: receiving at least one LiDAR detection data and at least one camera image data, wherein each of the at least one LiDAR detection data corresponds to at least a lower body portion of a target person, and wherein each of the at least one camera image data corresponds to at least an upper body portion of the target person; based on the at least one LiDAR detection data, determining at least one first positional output corresponding to the target person, wherein the at least one first positional output comprises at least: a first position of the target person and a corresponding first covariance matrix; based on the at least one camera image data, determining at least one second positional output corresponding to the target person, wherein the at least one second positional output comprises at least: a second position of the target person and a corresponding second covariance matrix; using a tracker module, determining an estimated position of the target person relative to a mobility device based on the at least one first positional output and the at least one second positional output.

[0006] According to another aspect, disclosed herein an autonomous mobility device comprising the above described system.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Various embodiments of the present disclosure are described below with reference to the following drawings:FIGs. 1A and 1B are images showing a target person, non-target persons, a mobility device, and a system for person following according to embodiments of the present disclosure;FIG. 2 is a parametric view of an autonomous mobility device and a system for person following according to various embodiments;FIG. 3 is a system diagram of a system for person following according to various embodiments;FIGs. 4A and 4B are exemplary diagrams showing a top view of an autonomous mobility device and a target person during person following according to various embodiments;FIG. 4C is an exemplary diagram illustrating the relative elevation (or position) of camera module and LiDAR modules according to various embodiments;FIG. 5 is a system diagram of another system for person following according to various embodiments;FIG. 6 is a system diagram of another system for person following according to various embodiments;FIG. 7 is a system diagram of another system for person following according to various embodiments;FIG. 8 is a system diagram of another system for person following according to various embodiments;FIG. 9 is a flowchart illustrating a method of person following according to various embodiments;FIG. 10 is a system diagram of an exemplary embodiment - Lidar-Depth Fusion Tracker (LDFT);FIG. 11 is a system diagram of an exemplary embodiment - Extended Lidar-Depth Fusion Tracker (XLDFT);FIG. 12 is a system diagram of an exemplary embodiment - Socially Aware Follow-Me (SAFM);FIG. 13 illustrate an exemplary implementation of a robotic wheelchair prototype with multiple cameras and LiDARs;FIGs. 14A and 14B show a comparison between a baseline (top row) and the exemplary implementation (bottom row). Output of the target person (in dotted line) being followed are shown as dotted line boxes, with FIG. 14B showing the ID switching to another person for the baseline (top row) while the target person with the same ID remained being tracked by the proposed method (bottom row);FIG. 15 illustrates a scenario where false positive track detected on the wall with an assigned track ID (in white). The baseline detects a person’s leg but also a false positive track on the wall as the pattern is very similar to human legs;FIG. 16 is a schematic diagram of a processor system.DETAILED DESCRIPTION

[0008] The following detailed description is made with reference to the accompanying drawings, showing details and embodiments of the present disclosure for the purposes of illustration. Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments, even if not explicitly described in these other embodiments. Additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0009] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0010] In the context of various embodiments, the term “about” or “approximately” as applied to a numeric value encompasses the exact value and a reasonable variance as generally understood in the relevant technical field, e.g., within 10% of the specified value.

[0011] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0012] As used herein, terms “concurrently”, “simultaneously”, “at the same time”, or the like, may refer to events or actions that coincide or overlap within a period of time, regardless of whether the events start at the same time instant, and regardless of whether the events end at the same time instant.

[0013] As used herein, the term “mobility device” may be used interchangeably with the terms “wheelchair”, “transport device”, “moving device”, “transport robot”, “robotic mobility device”, etc. and may generally refer to a device for moving or transporting a user, such as an elderly or an immobile or partially immobile user.

[0014] Conventional approaches towards mobility devices, such as wheelchairs, often include a caregiver or a companion pushing the wheelchair to move between locations. This requires the caregiver to stay behind the wheelchair, reducing the ability of the caregiver to monitor and to talk to the wheelchair user. Autonomous mobility devices, such as autonomous wheelchairs, represent a transformative leap in assistive technology, offering elevated levels of independence and safety for individuals with severe motor impairments. In addition, autonomous mobility devices alleviate the need for the caregiver or companion to be positioned behind the wheelchair, which allows the caregiver or companion to walk alongside or nearby to the autonomous wheelchair user during movement. This allows automated person following to be performed by the autonomous wheelchair, which allows the autonomous wheelchair to follow the caregiver during movement, enabling side-by-side monitoring and communication between the caregiver and the user.

[0015] In a hospital setting, the person following function in an autonomous wheelchair is especially desirable due to the ability to alleviate the physical / mental burden on a caregiver (or a medical staff) in pushing a wheelchair, while keeping the caregiver within a proximity of the autonomous wheelchair. The person following function may be employed in a majority of the commuting time and only disabled for more difficult navigation tasks such as entering an elevator, which may be enabled by a collaborative effort between the caregiver and the robot. Such burden sharing arrangement allows for the caregivers and / or medical staffs to focus on other higher-priority tasks.

[0016] Various challenges are present in person following implementations. One key challenge in person following is ensuring and enabling the autonomous wheelchair to reliably detect and track the target person in unstructured environments, such as rapid movement of multiple persons and / or changing environments. It was noted that the processes of detecting, tracking and following a target person are often met with tracking inaccuracies due to false positives introduced by other persons in the vicinity, changing environmental artefacts, inaccuracy in sensor data, etc. As an example, such false positives may include chairs being mistaken for legs of the caregiver.

[0017] As shown in FIGs. 1A and 1B, disclosed herein an autonomous mobility device 80 (or autonomous wheelchair) for performing person following. The mobility device 80 may comprise a system 100 for autonomously tracking a target person 84 (such as the caregiver or companion), allowing the autonomous wheelchair 80 to move in tandem or together with the target person 84. This beneficially enables the wheelchair user 82 to communicate with the caregiver 84, allowing more effective communication with and monitoring of the wheelchair user 82.

[0018] In various embodiments, in order to enable or to maintain proper contact between the caregiver(s) 84 and the wheelchair user 82, the ability to closely follow a person (or to performperson following) from one or more of: the front, the side or the back is important. To facilitate person following, in addition to manoeuvring the autonomous wheelchair 80 in a variety of environment / landscape, the autonomous wheelchair 80 maybe configured to track the caregiver 84 during movement. Referring again to FIGs. 1A and 1B, as a challenge to the person following process, there may be multiple non-target persons 86 / 88 (such as other pedestrians) present in the vicinity of the autonomous wheelchair 80 during commute. It is therefore important for the autonomous wheelchair to identify and follow the target person or caregiver 84 instead of other non-target persons 86 / 88 during movement, i.e. avoiding person switching during the commute.

[0019] It is non-trivial for the autonomous mobility device 80 to maintain person following in a variety of environments. It is especially so with multiple non-target persons around the mobility device 80. It was found that sensor data from conventional 2D LiDAR scans are insufficient for reliable person detection and tracking in various scenarios, such as a wider field of view. According to various embodiments, the autonomous wheelchair 80 may be configured to track and locate the caregiver 84 from a plurality of persons (including both target and non-target persons) during movement. In various embodiments, the proposed system 100 utilises multimodal sensor modules or multimodal sensing data for tracking the caregiver(s) 84 during movement. In an implementation, the system 100 may utilise LiDAR sensor modules in addition to camera modules, a combination which enables more information to be captured for enabling robust target person tracking and following.

[0020] The proposed system 100 may utilize both LiDAR sensor modules as well as camera modules in achieving a hybrid approach combining both a position-based approach corresponding to LiDAR sensors and an image-based approach corresponding to imaging cameras. The hybrid approach may facilitate the determination of an estimated position of a target person in 3D space at every time step. This may be enabled by tracking the person usingExtended Kalman Filter based trackers, in addition to deriving motion cues from image observations such as detection bounding boxes, velocity profile, IOU etc to determine a current state of tracked person. Such hybrid approach combines the advantages of both position-based and image-based approaches, in avoiding misidentification of the target person and potentially ID switching of the target person with a non-target person during person following.

[0021] According to various embodiments, disclosed herein is an autonomous mobility device for person following. The autonomous mobility device may be provided or integrated with a system for person following. The system may be configured to perform a method of person following.

[0022] FIG. 2 illustrates an autonomous mobility device 80 comprising a system 100 for person following, according to various embodiments of the disclosure. As an exemplary implementation, the mobility device 80 may be embodied as an autonomous wheelchair. In other implementations, the mobility device 80 may be a rehabilitation device, such as an assistive robot. The system 100 may comprise a processor 900 for performing a method of person following.

[0023] In various embodiments, the mobility device 80 may further comprise a LiDAR module 200 (Light Detection and Ranging module). The LiDAR module 200 may define a 360-degree LiDAR coverage around the mobility device 80. The LiDAR module 200 may comprise one or more LiDAR sensors 210 for capturing LiDAR detection data of objects (such as human subjects) around the mobility device 80. The LiDAR module 200 (or LiDAR sensors 210) may be configured to capture an all-around LiDAR measurement of the mobility device 80. For clarity, an “all-around LiDAR measurement” may comprise but is not limited to a 360-degree angle of LiDAR measurement around the mobility device 80 about a LiDAR measurement plane parallel to the ground. As examples, the LiDAR module 200 may comprise three LiDAR sensors 210 positioned around the mobility device 80, with each LiDAR sensor 210 configuredwith at least a 120-degree sensing angle. In other examples, the sensing angle of the LiDAR sensors 210 may overlap one other. In yet other examples, the LiDAR sensor 210 may comprise a larger sensing angle than 120 degrees, such as 180 degrees or 360 degrees.

[0024] In various embodiments, the mobility device 80 may further comprise a camera module 300. The camera module 300 may define a 360-degree field of view around the mobility device 80. The camera module 300 may comprise one or more imaging cameras 310 for capturing camera image data of objects (such as human subjects) around the mobility device 80. The camera module 300 (or imaging cameras 310) may also be configured to capture an all-around camera measurement of the mobility device 80. Similarly, an “all-around camera measurement” may comprise but is not limited to a 360-degree angle of camera measurement around the mobility device 80 about a camera measurement plane parallel to the ground. As examples, the camera module 300 may comprise three imaging cameras 310 positioned above the mobility device 80, with each imaging camera 310 configured with at least a 120-degree angle of view (or field of view). In other examples, the angle of view of the imaging camera 310 may overlap one other. In yet other examples, the imaging camera 310 may comprise a larger angle of view than 120 degrees, such as 180 degrees.

[0025] In various embodiments, the camera module 300 may further comprise a preprocessing module in which the camera images obtained from (or captured by) the imaging cameras 310 may be stacked or stitched to produce a single camera image. In further embodiments, the pre-processing module may also perform various imaging calibration or preprocessing processes, such as distortion adjustments / corrections, image cropping, adjusting for exposure / contrast / etc., which aid to facilitate the stacking or stitching of the camera images.

[0026] According to various embodiments, referring to FIG. 4C, the LiDAR sensors 210 of the LiDAR module 200 may be positioned elevated from the ground to capture LiDAR detection data from a lower body portion (such as a lower limb) of a target person. In addition,the camera module 300 may be positioned elevated from the main structure of the mobility device 80 to capture camera image data corresponding to at least an upper body portion of the target person. As such, the LiDAR module 200 and the camera module 300 may be positioned at different elevations relative to the mobility device 80.

[0027] FIG. 3 illustrates a system architecture of the proposed system 100 according to various embodiments of the disclosure. FIGs. 4A and 4B are schematic top views illustrating a method of person tracking performed by the autonomous mobility device 80 to track a target person 84 among other subjects 86 / 88. According to various embodiments, the system 100 may track the target person 84 and control the mobility device 80 to follow the target person 84 during a commute. In various scenarios as shown in FIG. 4A, the system 100 may control the mobility device 80 to follow beside (or side follow) the target person 84. In other scenarios as shown in FIG. 4B, the system may control the mobility device 80 to follow behind (or back follow) the target person 84.

[0028] In various embodiments, the system 100 may comprise a first object detection module 110. The first object detection module 110 may be configured to receive at least one LiDAR detection data 202 from the LiDAR module 200. Each of the at least one LiDAR detection data 202 may correspond to at least a lower body portion of a target person 84. As an example, the lower body portion may correspond to a lower limb of the target person 84.

[0029] In various embodiments, based on the at least one LiDAR detection data 202, the first object detection module 110 may determine at least one first positional output 112 corresponding to the target person 84. The at least one first positional output 112 may correspond to a predicted or estimated positional information of the target person 84 relative to the mobility device 80. In various embodiments, the at least one first positional output 112 may comprise at least: a first position of the target person 84 and a corresponding first covariance matrix. The first position of the target person 84 may be a first predicted position of the targetperson 84 relative to the mobility device 80. The first covariance matrix may correspond to a variance (or uncertainty) in the first position (or first predicted position) of the target person 84. In an example, the first positional output 112 may include a coordinate of a position, x,y coordinate bounding boxes, covariances, etc. The first object detection module 110 may also assign an ID to the target person 84 as well as to other non-target persons 86 / 88 using the at least one LiDAR detection data 202.

[0030] In various embodiments, the first object detection module 110 may be an object detection algorithm, comprising one or more of: a Leg detector ROS package for detecting human legs in 2D lidar scans or a Deep learning-based leg detectors.

[0031] In various embodiments, the system 100 may further comprise a second object detection module 120. The second object detection module 120 may be configured to receive at least one camera image data 302. Each of the at least one camera image data 302 may correspond to at least an upper body portion of the target person 84. As an example, the upper body portion may correspond to an image of the target person 84 above the chest.

[0032] In various embodiments, based on the at least one camera image data 302, the second object detection module 120 may determine at least one second positional output 122 corresponding to the target person 84. The at least one second positional output 122 may correspond to a predicted or estimated positional information of the target person 84 relative to the mobility device 80. In various embodiments, the at least one second positional output 122 may comprise at least: a second position of the target person 84 and a corresponding second covariance matrix. Similarly, the second position of the target person 84 may be a second predicted position of the target person 84 relative to the mobility device 80. The second covariance matrix may correspond to a variance (or uncertainty) in the second position (or second predicted position) of the target person 84. In an example, the second positional output122 may include a coordinate of a position, x,y coordinate bounding boxes, covariances, etc.

[0033] In various embodiments, the second object detection module 120 may be a machine learning-based module comprising one or more of: a YOLO model, a CenterNet model, a RF-DETR model, a DETR model, etc.

[0034] In various embodiments, the system 100 may also comprise a tracker module 140, such as a multi-object tracker module. The tracker module 140 may be configured to determine an estimated position 142 of the target person 84 relative to the mobility device 80 based on the at least one first positional output 112 and the at least one second positional output 122.

[0035] Still referring to FIG. 3, in various embodiments, the system 100 may further comprise a sensor fusion module 130. The sensor fusion module 130 may receive the at least one first positional output 112 and the at least one second positional output 122 from the first object detection module 110 and the second object detection module 120 respectively. In addition, the sensor fusion module 130 may perform sensor fusion to the at least one first positional output 112 and the at least one second positional output 122, to obtain a fused positional output 132. The fused positional output 132 may correspond to the target person. The fused positional output 132 may correspond to an estimated position 142 of the target person 84 based on a fusion between the at least one first positional output 112 and the at least one second positional output 122. As such, it may be said that the fused positional output 132 may be a combination of or a merger between the at least one first positional output 112 and the at least one second positional output 122. The fused positional output 132 may comprise at least: a fused position of the target person 84 and a corresponding fused covariance matrix. The fused position of the target person 84 may be a predicted position of the target person 84 relative to the mobility device 80. The fused covariance matrix may correspond to a variance (or uncertainty) in the predicted position of the target person 84. According to various exemplary embodiments, the sensor fusion of the at least one first positional output 112 and the at least one second positional output 122 may be performed using one or a combination of the followingmethods, but is not limited thereto: i) Kalman filters - all variants like KF, EKF, UKF, ii) Particle filters, iii) Feature matching, iv) Data association using Hungarian algorithm, used with Kalman filters, v) Data association by projecting lidar points to camera image, vi) Averaging / Central limit; PCA, etc.

[0036] In addition to the at least one first positional output 112 and the at least one second positional output 122, the fused positional output 132 may also be provided to the tracker module 140. Therefore, the tracker module 140 may be configured to determine an estimated position 142 of the target person 84 relative to the mobility device 80 based on the at least one first positional output 112, the at least one second positional output 122 and the fused positional output 132. In various embodiments, referring to FIGs. 4A and 4B, the estimated position 142 of the target person 84 relative to the mobility device 80 may comprise a 2-dimensional data, such as a top view.

[0037] In an exemplary embodiment, the tracker module 140 may be a modified UCMCTrack module. It is worth noting that in the proposed system and method, the UCMCTrack may be modified to track in Bird’s-Eye- View (BEV) plane (see FIGs. 4A and 4B) by using actual camera parameters, which are available. This departs from conventional methods, which assume tracking in a 2D homographic projective plane based on estimated camera parameters (intrinsic & extrinsic) from a video dataset.

[0038] In various embodiments, the mobility device 80 may further comprise a mobility device control module 150. The mobility device control module 150 may receive the estimated position 142 of the target person 84, and determine a navigation path of the mobility device 80 based on the estimated position 142 of the target person 84.

[0039] The method of person following may be performed in an iterative loop such that the mobility device 80 may continuously follow the target person in a commute. In an example, at a first time instance, a LiDAR detection data 202 and a camera image data 302 may be used todetermine an estimated position 142 of the target person 84 at the first time instance. Subsequently, in a second time instance, another LiDAR detection data 202 and another camera image data 302 may be used to determine another estimated position 142 of the target person 84 at the second time instance. As such, the iterative updating of the estimated position 142 of the target person 84 may be provided to the mobility device control module 150 to determine the navigation path of the mobility device 80.

[0040] FIG. 5 illustrates a system diagram of the proposed system 100 according to various embodiments of the present disclosure. Similar to previous embodiments, the system 100 may comprise a first object detection module 110, a second object detection module 120, and a sensor fusion module 130. Referring again to FIGs. 4A and 4B, in the scenario where there are a plurality of subjects (such as: the target person 84 and one or more non-target persons 86 / 88) in the vicinity of the mobility device, the system 100 may determine a respective estimated track of each of the plurality of subjects 84 / 86 / 88 in order to identify the target person 84 and to determine the estimated position 142 of the target person 84.

[0041] In various embodiments, the system 100 may further comprise a multi-object tracker module 140 incorporated with a track management module 160 for identifying the target person 84 from a plurality of subjects 84 / 86 / 88. Alternatively, the track management module 160 may be implemented independently from the tracker module 140. In various embodiments, the track management module 160 may identify the target person 84 from the plurality of subjects based on a tracking output of the tracker module 140. The tracking output may comprise a plurality of estimated tracks of the plurality of subjects 84 / 86 / 88. The track management module 160 may be configured to smooth the tracking output using a Kalman filter. Further, the track management module 160 may determine the estimated position 142 of the target person 84 relative to the mobility device 80 based on the estimated track of the target person 84.

[0042] In the scenario, in addition to the target person 84, the first object detection module110 may also determine respective first positional outputs corresponding to the other non-target persons 86 / 88. Similarly, the second object detection module 120 may also determine respective second positional outputs corresponding to the other non-target persons 86 / 88. Additionally, the fused positional outputs corresponding to the other non-target persons 86 / 88 may also be determined. The first, second and fused positional outputs corresponding to the other non-target persons 86 / 88 may similarly be provided to the tracker module 140 such that respective estimated tracks of the other non-target persons 86 / 88 are also determined. In this way, each of the persons (target or non-target) 84 / 86 / 88 are tracked by the system 100 concurrently.

[0043] In an exemplary embodiment, the track management module 160 may maintain an internal list of tracks as a list of KalmanBoxTracker objects. KalmanBoxTracker are KF objects for tracking a 4-dimensional state [x, v_x, y, v_y] that contains the position of points (x, y) and the respective velocities (v_x, v_y), z coordinate is omitted as only the 2D space is used for navigation. Object or subject motion may follow a constant velocity model and (x, y) coordinates may be taken as the measurement vector. The KalmanBoxTracker may also contain additional attributes, apart from the general KF implementation, such as: i) Name: name of the sensor {‘camera’, ‘lidar’, ‘camera+lidar’}; ii) Conf: confidence score of the detection; iii) Cam id: the id of the camera that detects; iv) Track id: the track id; v) Bbox coordinates:[bb lt x, bb lt y, bb rb x, bb rb y] that is the x, y coordinates of the top-left and bottomright of the bounding box, respectively; vi) Heading: an attribute given only to track that is of kind=TARGET; vii) Kind: whether a track is a IS TARGET or IS NOT TARGET; viii) death count: a counter for the age of the track.

[0044] The track management module 160 may iteratively update the internal list of tracks (self.trackers), as follows: I) Perform KF predict step for all KalmanBoxTracker in self.trackers; II) Perform ID matching by comparing the track id of each track in self.tracks to the list oftracks output by UCMCTrack (as input to the update function). The result of ID matching is a matrix of size (n matches, 2), containing the indices in self.trackers and the input list of tracks that correspond to the matches; III) Perform KF update step for all tracks in self.trackers, resets death count to 0; IV) Return newly updated tracks, i.e. tracks with death count < 3; V) Delete older tracks in self.trackers by traversing the list, incrementing death count every track by 1 and remove track with death count greater than a predefined maximum track age; VI) Create a list of unmatched detections from the list of the indices of detections that are not present in the matrix of matches; VII) Initiate new tracks, and add to the list self.trackers, using the list of unmatched detections.

[0045] In various instances, false positive tracks may be present during the person following tasks. False positive tracks may be lidar tracks that are initialized by the first object detection module 110 (or leg tracker) corresponding to artefacts that resemble human legs. This problem arises in office or lab environments, such as those with a number of chairs. During movement of the mobility device 80, the false positive tracks formed by the leg tracker may interfere with the assignment of estimated tracks. This problem is more severe when the mobility device 80 is moving or rotating due to the performance of multi-object tracking with respect to a moving coordinate frame (of the mobility device 80). This causes the reference points to appear to be moving, despite the objects (such as chairs) being stationary, which in turn causes the leg tracker algorithm not to be able to differentiate these false tracks (chairs) to actual targets to be tracked (human legs). Such false positive tracks may be mitigated effectively using multimodal input (i.e. camera + lidar).

[0046] As examples, the tracker module 140 and the track management module 160 may be implemented as one or more of: Kalman Filter (filter state and covariance) with HungarianMatching, KF -based 2D image-based trackers such as Deepsort, ByteTrack, OC-SORT, Deep- OC-SORT, BoT-SORT, StrongSORT, HybridSORT, with modifications to track in 2D bird’seye view (BEV) plane, Hybrid trackers such as Kalman Filter (filter state and covariance) with GNN / Transformer for ID matching.

[0047] FIG. 6 illustrates a system diagram of the proposed system 100 according to various embodiments of the present disclosure. In various embodiments, the second object detection module 120 may be configured to determine a plurality of intermediate second positional outputs based on a plurality of camera image data 302. The plurality of camera image data 302 may correspond to different fields of view of the target person 84 obtained concurrently. In other words, the plurality of camera image data 302 may comprise images of the target person 84 from different angles. Each of the plurality of intermediate second positional outputs may correspond to a positional output of the target person 84 for each of the plurality of camera image data 302. As such, each camera image data 302 with the target person 84 therein will correspond to a respective intermediate second positional output. In various embodiments, the second object detection module 120 may determine the at least one second positional output 122 by comparing an image boundary of respective ones of the plurality of camera image data.

[0048] In addition to the embodiments of FIG. 6 as previously described, the system 100 may utilize depth information to determine or recover the at least one second positional output 122 of the target person 84 from the second object detection module 120. In various embodiments, the camera module 300 may be configured to determine a plurality of depth data based on the plurality of camera image data 302. Alternatively, the camera module 300 may comprise at least one RGB camera and at least one depth camera. In addition, the second object detection module 120 may be further configured to determine the at least one second positional output 122 from the plurality of intermediate second positional outputs, by comparing the plurality of intermediate second positional outputs to a respective one of a plurality of depth data 304. As such, the estimated intermediate second positional output may be compared with the corresponding depth data 304. In this way, the plurality of intermediate second positionaloutputs are checked and compared to respective one of a plurality of depth data 304, in determining the at least one second positional output 122, improving the certainty thereof.

[0049] According to various embodiments, sensor fusion may first be performed by associating the plurality of camera image data 302 (or camera-camera data), followed by the camera image data 302 and the LiDAR detection data 202 (camera-lidar data). The association may check that the depth data 304 of a person obtained from both LiDAR detection data 202 and the depth data 304 derived from the camera image data 302 (after camera association which resembles a person) are approximately the same or corresponds to one another, thereafter recognizing the person as a positive case and passing on the information to the tracker module 140 for tracking with ID.

[0050] FIG. 7 illustrates a system architecture of the proposed system 100 according to various embodiments of the disclosure. In various embodiments, the system 100 may comprise a first object detection module 110, such as a leg tracker. The first object detection module 110 may be configured to receive a plurality of LiDAR detection data 202 / 222 from a first LiDAR module 200 and a second LiDAR module 220. The first object detection module 110 may receive at least one first LiDAR detection data 202 from the first LiDAR module 200, and at least one second LiDAR detection data 222 from the second LiDAR module 220. Each of the plurality of LiDAR detection data 202 / 222 may correspond to at least a lower body portion of a target person 84. The plurality of LiDAR detection data 202 / 222 may correspond to a plurality of lower limb portions of the target person 84. The plurality of lower limb portions may correspond to different elevations relative to the mobility device or relative to the ground.

[0051] In various embodiments, referring again to FIG. 4C, the first LiDAR module 200 may be positioned generally below a knee level (or a lower leg portion) of the target person 84, and the second LiDAR module 220 may be positioned generally above the knee level (or an upper leg portion) of the target person 84. As such, as an example, the at least one first LiDARdetection data 202 may correspond to a lower leg portion of the target person 84, and the at least one second LiDAR detection data 222 may correspond to an upper leg portion of the target person 84.

[0052] In various embodiments, the first object detection module 110 may determine a plurality of first positional outputs 112 / 114 corresponding to the target person 84 based on the plurality of LiDAR detection data 202 / 222. Each of the plurality of first positional outputs 112 / 114 may correspond to a predicted or estimated positional information of the target person 84 relative to the mobility device 80. In various embodiments, each of the plurality of first positional outputs 112 / 114 may comprise at least: a first position of the target person 84 and a corresponding first covariance matrix. The first position of the target person 84 may be a first predicted position of the target person 84 relative to the mobility device 80. The first covariance matrix may correspond to a variance (or uncertainty) in the first position (or first predicted position) of the target person 84.

[0053] In various embodiments, the system 100 may further comprise a second object detection module 120. The second object detection module 120 may be configured to receive at least one camera image data 302. Each of the at least one camera image data 302 may correspond to at least an upper body portion of the target person 84. As an example, the upper body portion may correspond to an image of the target person 84 above the chest.

[0054] In various embodiments, based on the at least one camera image data 302, the second object detection module 120 may determine at least one second positional output 122 corresponding to the target person 84. The second object detection module 120 may also be configured to perform facial recognition on the at least one camera image data 302 to identify the target person 84. The second object detection module 120 may further comprise a mapper module for identifying the target person 84. The at least one second positional output 122 may correspond to a predicted or estimated positional information of the target person 84 relative tothe mobility device 80. In various embodiments, the at least one second positional output 122 may comprise at least: a second position of the target person 84 and a corresponding second covariance matrix. Similarly, the second position of the target person 84 may be a second predicted position of the target person 84 relative to the mobility device 80. The second covariance matrix may correspond to a variance (or uncertainty) in the second position (or second predicted position) of the target person 84.

[0055] In various embodiments, the system 100 may further comprise a LiDAR fusion module 115. The LiDAR fusion module 115 may perform LiDAR fusion to the plurality of first positional outputs 112 / 114 to obtain a fused LiDAR output 117. The fused LiDAR output 117 may comprise at least: a fused position of the target person 84 and a corresponding fused covariance matrix. The fused LiDAR position of the target person 84 may be a predicted position of the target person 84 relative to the mobility device 80. The fused covariance matrix may correspond to a variance (or uncertainty) in the predicted position of the target person 84.

[0056] Still referring to FIG. 7, in various embodiments, the plurality of first positional outputs 112 / 114, the at least one second positional output 122, and the fused LiDAR output 117 may be provided to a tracker module 140. The tracker module 140 may be configured to determine the estimated position 142 of the target person 84 relative to the mobility device 80 based on the plurality of first positional outputs 112 / 114, the at least one second positional output 122, and the fused LiDAR output 117.

[0057] In various embodiments, the system 100 may determine the estimated position 142 of the target person 84 relative to the mobility device 80 based on a depth data 304 corresponding to the target person 84. The depth data 304 or depth information may be used to recover the 3D position (in which z coordinate may be later omitted) and / or 2D position of the target person 84 from the bounding box detected by the first / second object detection modules110 / 120. The recovery may be performed using camera deprojection (i.e. obtaining an inverseof the camera matrix), in which the camera intrinsics (such as device-specific parameters) and camera extrinsics (such as camera location and orientation) of each imaging camera may be known from the pinhole camera equation.

[0058] During initialization of tracks in the tracker module 140 (i.e. the modified UCMCTrack module), detections of type ‘camera+lidar’ (or fused positional output) are formed during the sensor fusion step. This may be possible when depth information is used to recover the 3D position from the camera images, in which will be fused with the 2D position (or first positional output) obtained from the 2D lidar. In some instances, the covariance matrix from the camera detection (or second positional output) may not be used during the sensor fusion (camera+lidar fusion). However, the camera detection (or second positional output) may be used to update the detection type to be ‘camera+lidar’ in addition to the camera id (the unique id of the camera that detects) and the bounding box information. For instances in which there is a need to recover a missing target person, in the context of ID recovery using facial recognition, may still require the recovery of the 3D position of the target person to reassign target ID.

[0059] Further referring to FIG. 8, in various embodiments of the present disclosure, the system 100 may further comprise a context-aware module 170. The context-aware module 170 may provide further feedback or guidance to the track management module 160 and / or the mobility device control module 150 to determine a navigation path of the mobility device 80 based on the estimated position 142 of the target person 84.

[0060] In various embodiments, the context-aware module 170 may determine an environment context of a surrounding of the mobility device 80. This aids the track management module 160 and / or the mobility device control module 150 to determine the navigation path based on the environmental context and the estimated position 142 of the target person 84.

[0061] In exemplary embodiments, the system 100 may control the mobility device 80 to side follow or back follow the target person 84 based on different contexts. In an example, when the mobility device 80 is navigating through a tight corridor, the environment context as determined by the context-aware module 170 switches the navigation path from a side following mode (Follow Me from side) to a back following mode (Follow Me from behind).

[0062] In another example, when the target person goes through a door and stops just outside the doorway, the mobility device 80 may end up occupying the doorway, blocking the passage. The context-aware module 170 may determine that this is an undesirable social situation, and control the mobility device 80 to move to another position.

[0063] In yet another door passing example, to go through a door with spring-loaded hinges, the target person 84 may hold the door to keep the door open. However, as the distance between the mobility device 80 and target person 84 may be less than a specified threshold, the mobility device 80 may remain stationary and does not move pass the door. With the context-aware module 170, the mobility device 80 may understand the intention of the target person 84 and may anticipate such actions and suitably adjusts the corresponding behaviour. As examples, the context-aware module 170 may comprise VLM models for scene understanding and reasoning, such as: Nvidia Cosmos Reasonl & Reason2.

[0064] In further embodiments, the context-aware module 170 may be provided with a voice input module 180. The voice input module 180 enables the translation of a user speech into commands for the mobility device 80 navigation. An example will be the integration of OpenAI Whisper to the context-aware module 170 to enhance Human Robot Interaction (HRI) translating of user speech into commands for mobility device 80 navigation.

[0065] In various embodiments, the context-aware module 170 may determine a behavioural context between a mobility device user 82 (or the user of the wheelchair) and the target person84. This enables the track management module 160 and / or the mobility device control module150 to determine the navigation path based on the behavioural context and the estimated position of the target person. For example, when the mobility device user 82 and the target person 84 are in a conversation, the system 100 switches the navigation path to a side following mode.

[0066] According to another aspect of the invention, referring to FIG. 9, disclosed herein is a method 700 of person following. The method 700 may comprise: in 710, receiving at least one LiDAR detection data and at least one camera image data, wherein each of the at least one LiDAR detection data corresponds to at least a lower body portion of a target person, and wherein each of the at least one camera image data corresponds to at least an upper body portion of the target person; in 720, based on the at least one LiDAR detection data, determining at least one first positional output corresponding to the target person, wherein the at least one first positional output comprises at least: a first position of the target person and a corresponding first covariance matrix; in 730, based on the at least one camera image data, determining at least one second positional output corresponding to the target person, wherein the at least one second positional output comprises at least: a second position of the target person and a corresponding second covariance matrix; and in 740, using a tracker module, determining an estimated position of the target person relative to a mobility device based on the at least one first positional output and the at least one second positional output.

[0067] In various embodiments, the method 700 further comprises: performing sensor fusion to the at least one first positional output and the at least one second positional output to obtain a fused positional output, wherein the fused positional output corresponds to the target person; and determining the estimated position of the target person relative to the mobility device, by providing each of the at least one first positional output, the at least one second positional output and the fused positional output to the tracker module.

[0068] In various embodiments, the method 700 further comprises: based on a plurality of LiDAR detection data, determining a plurality of first positional outputs corresponding to the target person; performing LiDAR fusion to the plurality of first positional outputs to obtain a fused LiDAR output; and providing the fused LiDAR output to the tracker module to determine the estimated position of the target person relative to the mobility device.

[0069] In various embodiments, the method 700 further comprises: determining a plurality of intermediate second positional outputs based on a plurality of camera image data, wherein the plurality of camera image data corresponds to different fields of view of the target person obtained concurrently; and determining the at least one second positional output from the plurality of intermediate second positional outputs by comparing the plurality of intermediate second positional outputs to a respective one of a plurality of depth data.

[0070] In various embodiments, the method 700 further comprises: determining the at least one second positional output by comparing an image boundary of respective ones of the plurality of camera image data. In various embodiments, the method 700 further comprises: determining the estimated position of the target person relative to the mobility device based on a depth data corresponding to the target person. In various embodiments, the method 700 further comprises: obtaining the at least one LiDAR detection data using a LiDAR module coupled to the mobility device, wherein each of the at least one LiDAR detection data corresponds to at least a lower limb of the target person; and obtaining the at least one camera image data using a camera module coupled to the mobility device, wherein each of the at least one camera image data corresponds to at least an upper body image of the target person.

[0071] In various embodiments, the method 700 further comprises: performing facial recognition on the at least one second positional output to identify the target person. In various embodiments, the method 700 further comprises: identifying the target person from a tracking output of the tracker module, wherein the tracking output comprises at least an estimated trackof the target person; and determining the estimated position of the target person relative to the mobility device based on the estimated track. In various embodiments, the method 700 further comprises: smoothing the tracking output using a Kalman filter.

[0072] In various embodiments, the method 700 further comprises: determining a navigation path of the mobility device based on the estimated position of the target person. In various embodiments, the method 700 further comprises: determining an environment context of a surrounding of the mobility device; and determining the navigation path based on the environmental context and the estimated position of the target person. In various embodiments, the method 700 further comprises: determining a behavioural context between a mobility device user and the target person; and determining the navigation path based on the behavioural context and the estimated position of the target person.

[0073] Exemplary Embodiment #1 - Lidar-Depth Fusion Tracker (LDFT)

[0074] FIG. 10 illustrates a system diagram of the exemplary embodiment of the proposed system - a Lidar-Depth Fusion Tracker (LDFT) implemented as a robotic wheelchair. LDFT comprises enhanced user interface with interactive real-time video feed, and is a cost saving alternative to 3D LiDAR systems by utilizing 2D LiDARs and ToF RGBD cameras. In addition, LDFT comprises higher tracking accuracy and reduced false positive rate, using LiD AR-camera sensor fusion, when compared to other single sensor methods.

[0075] LDFT was implemented in a robotic wheelchair research platform with the following sensors: 1. ToF RGBD camera: 2 x Percipio TM266-E1 (mounted at height ~1.73m from wheelchair base) 2. 2D lidar: 3 x Pacecat LDS50C-E (front left, front right, rear at height ~0.16m from wheelchair base).

[0076] To improve person tracking accuracy for person-following mobile robotic wheelchair, main steps involved in LDFT are: a) Detection and tracking of leg pairs around the wheelchair are performed using 2D LiDAR range sensing at a height below knee level. Theposition of the detected person is determined by the midpoint of the tracked leg pairs. This functionality is provided by a leg tracker package. The output will consist of LiDAR detections, representing the detected person’s 3D position relative to the wheelchair, b) Detection of people around the wheelchair using an object detection model such as the YOLO model from image sensing. Depth sensing is then used to obtain the position of detected person in 3D coordinates relative to the wheelchair, c) Sensor fusion is performed between detections from the camera array to merge detections of the same person appearing in adjacent cameras, caused by overlapping FOVs; If a person appears in two bounding boxes from adjacent cameras, the larger bounding box is selected. This will output a set of unique camera detections for each person, d). Sensor fusion is then performed between LiDAR and camera detections to reduce measurement errors, e) The output from sensor fusion, containing both fused and unfused detections, is input into a modified multi-object tracker. Since the z-coordinate of all 3D detections is omitted (assuming no inclination), the tracking operates as 2D tracking on the bird’s-eye-view plane around the wheelchair. The tracker employs non-IOU-based ID association, such as UCMCTrack, which uses Mapped Mahalanobis Distance for track ID association, and was utilized in the exemplary embodiment.

[0077] Exemplary Embodiment #2 - Extended Lidar-Depth Fusion Tracker (XLDFT)

[0078] FIG. 11 illustrates a system diagram of the exemplary embodiment of the proposed system - an Extended Lidar-Depth Fusion Tracker (XLDFT) implemented as a robotic wheelchair. XLDFT has a higher tracking accuracy compared to baseline LDFT and may enhance the reliability of LDFT, with additional lidar-lidar sensor fusion. XLDFT includes appearance-based matching to recover long lost target e.g. facial recognition. XLDFT comprises the features of: a) Integration of another 2D LiDAR range sensing at height above knee level, b). Sensor fusion between LiDAR detections from two different levels, i.e. first level: below the knee, second level: above the knee, c) The sensor fusion output, containingonly LiDAR detections, is input into a modified TBD multi-object tracker, d) Integration of facial recognition module as a plugin to process RGBD images from camera to recover lost target.

[0079] Exemplary Embodiment #3 - Socially Aware Follow-Me (SAFM)

[0080] FIG. 12 illustrates a system diagram of the exemplary embodiment of the proposed system - a Socially Aware Follow-Me (SAFM) implemented as a robotic wheelchair. SAFM comprises: i) Intuitive Follow-Me mode switching between automatic and manual operation. This allows users to takeover tasks that the robot cannot do ensuring smooth human-robot collaboration, ii) Context-aware person following using vision-based Al models to improve navigation and social compliance. By understanding the user’s intention through scene understanding (crowds, gestures, intent) the robot behavior can be adapted accordingly, iii) Enhanced user experience with HRI. Social alarm trigger via sound and voice responses e.g. emergency button pressed, requesting path clearance to people nearby, person tracking failure, etc. iv) Safety enhancement with automatic obstacle detection and collision avoidance, v) Robust follow-me. By using LiDAR scan and generated context from the cameras, information of the robot’s surroundings such as detected obstacles, narrow passages and social contexts, are utilized to enable automatic switching of the Follow -Me mode, from side following to following from behind and vice versa. This allows the Follow -Me to operate in either following from side or from behind the target person depending on the robot’s surroundings. Moreover, this allows a more seamless and adaptive navigation experience. The Social Follow-Me system integrates LDFT variants and SAFM to create a holistic solution for the Follow -Me function to be implemented on electrically powered motorized wheelchairs, improving hospital workflow and assisting caregivers.

[0081] SAFM may increase social interaction quality between the robotic wheelchair and people around the wheelchair: a) Intuitive toggling between manual or Follow-Me mode ofoperation e.g. on press of a touchscreen or a physical button, b) Two modes of Follow-Me function are available: 1) goal mode that require prior map and a goal to be specified by the user for intention prediction for better path planning and obstacle avoidance, 2) no-goal mode, no goal and no prior map required, without intention prediction, c). Automatic path planning and navigation using ROS1 / ROS2 navigation stack.

[0082] d). A monitor to display current target person and his / her surroundings e.g. LCD.The target person initiation can be done as follows: i. User enables Follow-Me function, ii. User press a physical button on the wheelchair e.g. a button on the left armrest, iii. Target is assigned to the user, can be seen as a green box enclosing the user on the display.

[0083] e) User interface for users to select operation mode e.g. touchscreen. A probable scenario for the usage of touchscreen: i. User selects goal mode. ii. User selects a destination in goal mode, e.g. emergency room. iii. Path planner plans the most feasible path. iv. Wheelchair aligns itself with the direction of the local path towards emergency room. v. Initiate navigation right after the target person starts moving.

[0084] f). Context-aware person following: i. Vision language models (VLMs) such asCLIP, Large VLMs (GPT-4 / Gemini) can be used to generate contexts from multimodal sensing to understand a wide range of contexts such as door passing scenarios as described above, ii. The contexts generated from VLMs are then passed to the motion planner such as Dynamic Window Approach (DWA) that performs simple collision avoidance and goal-reaching behaviors.

[0085] g) Robust follow-me: i. Leveraging 2D LiDARs (360° coverage) and wide FOV cameras (RGB Horizontal / Vertical: 97° / 65° each) enhances scene understanding, obstacle awareness, and path planning for VLM-based navigation. For example, H: 110° and H:120°FOV cameras respectively for socially-aware robot navigation.

[0086] Exemplary Implementation - Robotic wheelchair prototype

[0087] The exemplary implementation comprises a position-based tracking approach using deep learning for person detection, multi-sensor fusion and modified UCMCtrack algorithm and deployed on a robotic wheelchair. The proposed sensor fusion approach integrates camera and LiDAR detections as probability distributions, considering both position (mean) and covariance matrix, which were useful for Kalman filter-based tracking. The covariance matrix was extended to polar coordinates, enabling fusion of detections from multiple sensors in 3D space, improving person detection and reducing false positives compared to 2D LiDAR tracking. Multiple cameras with overlapping fields of view were used to enable better track initiation and to allow the user to be detected anywhere around the robot, alongside LiDAR.

[0088] Additionally, modification was made to UCMCtrack, which efficiently tracks objects by projecting detections onto the ground plane (obtained from image) as probability distributions and applying a Kalman filter with Mapped Mahalanobis Distance (MMD) for improved tracking accuracy. By combining UCMCtrack with the proposed sensor fusion framework, robust human tracking in 3D space using multiple sensing modalities was achieved. In order to compare position-based tracking approach with image-based tracking, OCSORT and Hybrid SORT were implemented using the wheelchair robot’s cameras. Experimental results showed that the position-based tracking performed much better than baseline approaches in terms of consistent ID retention, association with fewer false positives.

[0089] The exemplary implementation computes the position and covariance matrix for both camera image data and 2D LiDAR detections, allowing the sensor fusion algorithm to decide whether to merge them. The fused detections were input into a modified UCMCtrack for tracking and following a person in a wheelchair.

[0090] Hardware Configuration

[0091] The proposed tracking framework was implemented, tested and verified in the wheelchair prototype as shown in FIG. 13. The prototype included 3 orbbec RGB-D camerason the top to cover a field of view of close to 200 degrees in front of the wheelchair. Also, 3 LiDARs are included to provide laser scan of the environment around the wheelchair. All required modules such as mapping, navigation and localization etc were ran in ROS framework. Given the wheelchair hardware, the aim was to develop a person / target person tracking system that may robustly track the person anywhere around the wheelchair. The tracking framework aimed to provide the position of the target with respect to a fixed world coordinate. It is thus crucial for the framework to provide a consistent tracking ID for the target person being followed without losing track of the target person.

[0092] Overview of the proposed framework

[0093] As shown in FIG. 13, the proposed framework was divided into three components, namely, person detections from camera and LiDAR; sensor fusion to identify the actual target person and a modified UCMCtrack that handled detections (fused and separate) to provide respective consistent tracking ID for the persons around the wheelchair.

[0094] For vision-based person detection, YOLO was used to identify all persons in the cameras around the wheelchair, assigning each detection a bounding box and confidence score. Depth dswas estimated using the bounding box coordinates in the depth frame, and the 3D world position was computed via inverse projection using the centroid, depth, and the cameras’ intrinsic and extrinsic parameters. The wheelchair’s base_link serves as the reference frame for the extrinsic matrix.

[0095] For LiDAR based person detection, the leg_tracker package was utilised, which in turn output the position (x,y) ∈ ℝ2for each detected pair of legs, a track ID and a confidence score for each detection. The position for each detection is again provided with respect to the wheelchair’s base_link.

[0096] A component of sensor fusion and tracking is the covariance matrix R ∈ ℝ2×2, which quantified the uncertainty of each detection’s position (x,y). Confidence scores from theYOLO and leg_tr acker are retained for camera and LiDAR detections, respectively. Camera-camera detection association is first performed to match detections from neighboring cameras, followed by LiDAR-camera detection association.

[0097] Finally, all detections, which were represented in the form of position (x, y) in the bird’s-eye view with an associated covariance matrix R defined with respect to the wheelchair’s basej nk were provided as input to UCMCTrack. Hereafter, person tracking and following were performed relative to wheelchair’s base_link.

[0098] Unified Representation for Person Detections

[0099] In the implementation, the camera image data and lidar detection data were unified under one representation:(x, y, R, conf, bbox\ [bb_lt_x, bb_lt_y, w, h], cam_id, fuse_state_name, ID) which comprises the position (x, y) ∈ ℝ2in 3D coordinates (z as height is omitted), covariance matrix R ∈ ℝ2×2, confidence score conf ∈ (0,1) ⊂ ℝ, bounding box information: top left pixel coordinates (bb_lt_x, bb_lt_y) ∈ ℤ≥0, width w ∈ ℤ≥0, height h ∈ ℤ≥0, camera ID cam_id ∈ ℕ to distinguish between camera detections from different cameras, fuse_state_name is the current fusion state of the detection. For example, ‘camera ‘lidar’, ‘ earner a+lidar’, and the track ID which was assigned once the detection has been through the tracker. Since the bounding box information and camera ID were applicable only to camera detections, for lidar detections, the bounding box information is filled with zeros i.e. bbox\ (0,0, 0,0) and cam_id with -1 i.e. cam_id: -1.

[0100] Computation of Detection Covariance Matrices

[0101] Covariances of detections are crucial for computing Mahalanobis distances which serve as the association metric employed in UCMCTrack. In conventional UCMCtrack framework, the tracking is done in a single camera frame and defines covariance matrices in terms of ground plane obtained from (u, v) pixel in 2D image space alone which is not valid orcompatible with the proposed approach in which both camera image data and 2D LiDAR detections are used.

[0102] To achieve a more generalized covariance matrix definition, a Gaussian distribution in polar coordinates was defined, which allowed the system to deal with both types of detection (camera and LiDAR), comparing the respective positions and fusing them if needed. In addition, this enabled tracking in 3D space around the wheelchair.

[0103] The wheelchair coordinate axes was defined as base_link (x’ y'), with basis vectors i' and ]', as the standard coordinate Cartesian coordinates (x, ), with basis vectors i andj, rotated by 90° counter-clockwise (CCW) around the z-axis. Here, the z-axis is defined as the direction perpendicular to the (x, y) plane in ℝ2, corresponding to the positive z direction in a right-handed coordinate system. Also polar coordinates (r, (p), with basis vectors and z where r ∈ ℝ≥0is the radial distance and the angle ∈ [−π, π] ⊂ ℝ is the angle measured from x’ axis to the position vector r = x'l' + y'j' positive CCW (following the right hand rule).

[0104] The covariance matrix of the Gaussian distribution may then be defined as:where

[0105] σrφ= σφrsince covariance matrices are required to be symmetric and positive definite. kr∈ ℝ>0and kφ∈ ℝ>0are the scaling factors that affects how the radial distance r and angle are related. p( ) is the correlation parameter that adjusts the covariance between variables r and <p.

[0106] To satisfy positive definiteness, the determinant of the covariance matrix should be greater than zero i.e. detΣ(r, φ) > 0. The condition,detΣfollows. Choosing ρ(φ) = ρ0cos(2φ) G (—1,1) c IR, where p0being the correlation coefficient, — 1 < p0< 1, and noting that cos(20) is symmetric along the range [—7i,7i], making it a suitable choice in consideration of the problem’s geometry.

[0107] For experiment sake, ρ0was chosen to be 0.5, kris chosen to be 0.05 for lidar detections and 0.07 for camera detections and kφis chosen to be 0.25 for both camera and lidar detections.

[0108] Each of the detections contained position (x, y) in 3D coordinates as z (height) was omitted, which was equivalent to 2D tracking. This was because tracking was done in the 2-dimensional Euclidean space ℝ2, and hence the covariance matrix should be transformed to the standard Cartesian coordinates (x,y).

[0109] Given the relationshipx' = y, y' = −x ⇒ x = −y', y = x'andx' = rcosφ, y' = rsinφthe Jacobian matrix J of the transformation (r, (f)) -> (%’, y') -> (x, y) is given byJx,y(r, 0) = Jx,y x y’ Jx.fy.(r, 0)Thus the covariance matrix Z(x,y) may be computed as:Σ(x, y) = Jx,y(r, φ)Σ(r, φ)Jx,y(r, φ)TIn which the expression may then be calculated numerically by substituting, r = √(x'2+ y'2) and φ = atan2(y', x'). Therefore, each detection may be written as det ~ 풩((x', y'), R), where (%',y') represents the mean and R = Σ(x, y) denoted the covariance matrix. It wasworth noting that Σ(x, y) and Σ(x', y') were related by a rotation about the z-axis. Specifically,Σ(x, y) may be obtained by applying a rotation to Σ(x', y') and vice versa, Σ(x', y') may be obtained by applying a −π / 2 rotation to Σ(x, y).

[0110] Sensor Fusion Algorithm

[0111] Association of Camera-Camera Detections

[0112] From the hardware shown in Fig. 12, it may be seen that there were multiple cameras with overlapping FOV between neighbouring cameras. Firstly, the aim was to associate detections of the same person, who may appear in two cameras due to the overlapping FOVs. A general setup of n E N cameras was considered, with n > 1 arranged radially facing outwards at distinct angles φk∈ (−π, π), where k is the camera number, k E {1,2,...,n} and is the angle between the x'-axis and the vector resulting from the projection of camera optical axis zconto the 2D plane ℝ2.

[0113] Define the sequence S of ordered pairs of consecutive camera IDs as follows:Each pair (I, I + 1) E S represents the ordered pair of consecutive camera IDs in sorted order of angles of the corresponding cameras. The sequence S captures the adjacency of camera IDs in decreasing order of their angles.

[0114] The association process involved traversing the sequence S, where for each pair (I, I + 1), detections from camera ID I are matched with those from camera ID Z + 1 based on the position of the detection. For n > 2, each element I E {2,3,...,n — 1} appeared twice in S, once in (Z — 1, Z) and once in (Z, Z + 1). Thus, care may be taken when associating detections between consecutive camera IDs, as unmatched detections from (Z — 1, Z) may be matched with detections from (Z, Z + 1).

[0115] The distance metric used to construct the association cost matrix for detections between two adjacent cameras, detaand detfo, was the Euclidean distance, deuclidean:where (xa, ya) refers to a detection with cam_id = l and (xb, yb) refers to another detection with cam_id = l + 1. It is important to note that not all detections present in the camera pair were included in the association process, but instead, only detections with bounding boxes intersecting the left or right border of the camera image frame in the camera pair (l, l + 1) were used.

[0116] Merging of Camera-Camera Detections

[0117] Merging of two camera detections, detaand detfo, may be done bywhere μfusedgeometrically means taking the midpoint of the line segment connecting μaand μbas the combined mean. While Σfusedis the covariance of the probability density function (pdf) resulting from the multiplication of two Gaussian pdfs.

[0118] Association of Camera-Lidar Detections

[0119] The resulting detections from the association of camera-camera detections were then used for camera-lidar detections association. The association cost matrix for assignments of camera-lidar detections was constructed based on the expected squared distance between two Gaussian distributions. Given two Gaussian pdfs, one of camera detection detc~ 7\f((xc,yc), / ?c) and the other of lidar detection detd~ 풩((xd, yd), Rd) that are independent, the expected squared distance is given bywhere the first term is the squared distance between the means and the second term is the trace of the sum of the covariance matrices.

[0120] Merging of Camera-Lidar Detections

[0121] Merging of camera and lidar detections may be performed either by using inversevariance weighting of the pdfs of the camera and LiDAR, where the weights were modulated by the respective confidence scores, or by relying solely on the LiDAR pdf. Based on qualitative observations, no significant difference were observed between the two approaches. Therefore, it was chosen to use only the LiDAR measurement, as it was also known to have higher positional accuracy compared to the camera.

[0122] Modified UCMCtrack

[0123] The tracking framework utilized fused ‘camera + lidar’ as well as separate ‘camera ’ and ‘lidar’ detections provided by sensor fusion. Each detection comprised position x,y in 3D space around the wheelchair and covariance matrix R, of which UCMCtrack used to compute MMD. Specifically, changes were made in the initialization of tracks and track management.

[0124] Initialization of Tracks

[0125] In the exemplary implementation of UCMCTrack, a detection was only initialized as a track when it was in the fused state or ‘camera+lidar’. This aided in reducing the number of false positive tracks and contributed to better tracking performance in an office environment where there were an abundant number of chairs. Upon initialization, while updating the tracks, the fuse_state_name of the detection were not considered. This was due to the need to track person(s) who exits the camera frames or LiDAR range, who still needed to be tracked.

[0126] Track Management

[0127] Track management is an auxiliary tracker that maintained an internal list of Kalman tracks with IDs that of the output from our implementation of UCMCTrack. The main purpose is to smoothen jaggy outputs from the tracker upstream, to manage tracks and to transform theknown relative position and heading direction of the target to a fixed world coordinate for person following. The proposed method was implemented with the wheelchair prototype for successful target following.

[0128] Experimental Results and Discussion

[0129] Dataset

[0130] For testing the efficiency of the proposed tracking framework in mobile robots, it was crucial to have a dataset that contained both multiple cameras and 2D LiDAR modalities. Also, it is required to have the ground truth location of people around and actual target whom the mobile robot is following. Thus, a dataset with multiple cameras and 2D LiDAR was collected using the wheelchair prototype. For data collection, rosbags with the required topics were collected. The rosbags were recorded with various conditions such as: one person, multiple persons and actual crowded scenarios. The ground truth bounding boxes were marked in each of the camera frames. During data collection, the wheelchair was manually controlled by an operator (using joystick) to follow the actual target. This was to ensure that our dataset is able to capture the person following mobile robot use case. A total 382.005596s of data was recorded.

[0131] Evaluation Metric

[0132] The HOTA metrics was employed to evaluate the effectiveness of the proposed method. Whenever a person is available in the camera frame, the bounding box is annotated manually. HOTA introduced a set of performance metrics that provided better understanding of tracker behavior in the Multi object Tracking (MOT) field. Using the metric, the effectiveness of the tracking ID association and better ID retention may be shown. Also, false positive rate was used to compare the proposed method and a baseline in terms of false positives performance.

[0133] Baseline

[0134] For the baseline, an image-based SORT approach and a position-based LiDAR tracking method were considered. Since position-based LiDAR tracking method deals only with LiDAR scans, direct estimation of HOTA cannot be performed. Therefore, sensor fusion with camera was performed using the proposed method. However, instead of using the tracking ID provided by the proposed method, the tracking ID provided by the position-based LiDAR tracking method was used. This allowed the ID association and retention capabilities of the baseline to be tested. For false positives, the tracks detected by the leg tracker without any sensor fusion were considered. The aim was to provide rigorous evaluation of these approaches in person following applications.

[0135] Results and Discussion

[0136] Better Tracking Performance

[0137] Referring to Table 1, it is clear that the proposed method was able to better track the person through both camera and 2D LiDAR. In addition to the baseline approach, the proposed method was also compared against popular vision-based tracking approaches OC-SORT and Hybrid SORT. HOTA measures how well the trajectories of matching detections align, and averages this over all matching detections, while also penalising detections that don’t match. From Table 1, it may be seen that the proposed method outperformed all other methods in the overall HOTA score. Comparing the metrics, DetA and AssA, it may be seen that proposed method was comparable to the baseline in terms of DetA, and outperformed baseline in AssA. A further observation is that for single person cases, the baseline is able to detect the target quite reliably even in the presence of false positives, however, this comes at the expense of crowd ID switching (person ID switching) and target loss, which happened frequency. From FIGs. 14A and 14B, it may be seen that the baseline (FIGs. 14A and 14B, top row) suffered from ID switching whereas the proposed method (FIGs. 14A and 14B, bottom row) was able to retain the ID even in a crowd.Table 1. Comparison of the HOTA metrics between proposed and baseline methods

[0138] Lower false positives

[0139] To evaluate false positives, rosbags were used with a single person walking in front of the wheelchair, thus causing only one valid track (for ground truth). False positives were determined by counting extra initiated tracks. Table 1 above shows the proposed method eliminated false positives by leveraging the camera’s semantic information, whereas the baseline method detected 23 false positives due to environmental artifacts in the LiDAR scan (See FIG. 15).

[0140] The proposed autonomous mobility device and the proposed system for performing a method of person following may be implemented by a processor system 900 as illustrated in the schematic block diagram of FIG. 16. Components of the processing system 900 may be provided within one or more computing device to carry out the functions of the modules or any other modules. One skilled in the art will recognize that the exact configuration or arrangement illustrated in FIG. 16 is provided by way of example only, e.g., each processing system provided may be different and the exact configuration of processing system 900 may vary.

[0141] In embodiments of the present disclosure, the processing system 900 may include a controller 901 and user interface 902. User interface 902 is configured to enable manual interactions between a user and the computing module as required. For this purpose, theprocessing system 900 includes the input / output components required for the user to enter instructions to provide updates to each of the modules. A person skilled in the art will recognize that components of user interface 902 may vary from embodiment to embodiment but may typically include one or more input devices 935 such as but not limited to a touchscreen, a keyboard, a joystick, a mouse, a microphone, etc. The user interface 902 can also include a media player 940, which can be in the form of one or more playback devices, including but not limited to a display, a speaker, earphones, headsets, etc.

[0142] The controller 901 is configured to be in data communication with the user interface 902 via bus 915. The controller 901 includes memory 920 and processor 905 mounted on a circuit board to process instructions and data, e.g., to perform the method of the present disclosure. The controller 901 includes an operating system 906, an input / output (I / O) interface 930 for communicating with user interface 902, and a communications interface, e.g., a network card 950. The network card 950 may, for example, be configured to send data from the controller 901 via a wired or wireless network to other processing devices or to receive data via the wired or wireless network. Wireless networks that may be utilized by the network card 950 include, but are not limited to, Wireless-Fidelity (Wi-Fi), Bluetooth, Near Field Communication (NFC), cellular networks, satellite networks, telecommunication networks, Wide Area Networks (WAN), and etc.

[0143] Memory 920 and operating system 906 are in data communication with central processing unit (CPU) 905 via bus 910. The memory 920 may include both volatile and nonvolatile memory. The memory 920 may include more than one of each type of memory, e.g., Random Access Memory (RAM) 923, Read Only Memory (ROM) 925, and a mass storage device 927. The mass storage device 927 may include one or more solid-state drives (SSDs).One skilled in the art will recognize that the memory described above includes non-transitory computer-readable media and shall be taken to include all computer-readable media except fora transitory, propagating signal. Typically, instructions are stored as program code in the memory but can also be hardwired. Memory 920 may include a kernel and / or programming modules such as a software application that may be stored in either volatile or non-volatile memory.

[0144] Herein, the term “processor” is used to refer generically to any device or component that can process computer-readable instructions, including for example, a microprocessor, microcontroller, programmable logic device, or other computational device. That is, processor 905 may be provided by any suitable logic circuitry for receiving inputs, processing them in accordance with instructions stored in memory, and generating outputs (for example to the memory components or media player 940). In the present disclosure, processor 905 may be a single core or multi-core processor with memory addressable space. In one example, processor 905 may be multi-core, comprising—for example—an 8 core CPU. In another example, it could be a cluster of CPU cores operating in parallel to accelerate computations.

[0145] Further, one skilled in the art will recognize that certain functional units in this description have been labelled as modules throughout the specification. The person skilled in the art will also recognize that a module may be implemented as circuits, logic chips or any sort of discrete component. Still further, one skilled in the art will also recognize that a module may be implemented in software which may then be executed by a variety of processor architectures. In embodiments of the disclosure, a module may also comprise computer instructions or executable code that may instruct a computer processor to carry out a sequence of events based on instructions received. In further embodiments, the module may comprise a combination of different types of modules or sub-modules. The choice of the implementation of the modules may be determined by a person skilled in the art and does not limit the scope of the claimed subject matter in any way.

[0146] All examples described herein, whether of apparatus, methods, materials, or products, are presented for the purpose of illustration and to aid understanding, and are not intended to be limiting or exhaustive. Modifications may be made by one of ordinary skill in the art without departing from the scope of the invention as claimed.

Claims

CLAIMS1. A system, comprising:memory storing instructions; anda processor coupled to the memory and configured to process the stored instructions to implement:a module configured to perform a method of person following, the method including:receiving at least one LiDAR detection data and at least one camera image data, wherein each of the at least one LiDAR detection data corresponds to at least a lower body portion of a target person, and wherein each of the at least one camera image data corresponds to at least an upper body portion of the target person;based on the at least one LiDAR detection data, determining at least one first positional output corresponding to the target person, wherein the at least one first positional output comprises at least: a first position of the target person and a corresponding first covariance matrix;based on the at least one camera image data, determining at least one second positional output corresponding to the target person, wherein the at least one second positional output comprises at least: a second position of the target person and a corresponding second covariance matrix; using a tracker module, determining an estimated position of the target person relative to a mobility device based on the at least one first positional output and the at least one second positional output.

2. The system as recited in claim 1, wherein the method further comprises: performing sensor fusion to the at least one first positional output and the at least one second positional output to obtain a fused positional output, wherein the fused positional output corresponds to the target person; and determining the estimated position of the target person relative to the mobility device, by providing each of the at least one first positional output, the at least one second positional output and the fused positional output to the tracker module.

3. The system as recited in any one of the above claims, wherein the method further comprises:based on a plurality of LiDAR detection data, determining a plurality of first positional outputs corresponding to the target person;performing LiDAR fusion to the plurality of first positional outputs to obtain a fused LiDAR output; andproviding the fused LiDAR output to the tracker module to determine the estimated position of the target person relative to the mobility device.

4. The system as recited in claim 3, wherein the plurality of LiDAR detection data corresponds to a plurality of lower limb portions of the target person, wherein the plurality of lower limb portions corresponds to different elevations relative to the mobility device.

5. The system as recited in claim 4, wherein the plurality of lower limb portions comprises a lower leg portion and an upper leg portion.

6. The system as recited in any one of the above claims, wherein the method further comprises:determining a plurality of intermediate second positional outputs based on a plurality of camera image data, wherein the plurality of camera image data corresponds to different fields of view of the target person obtained concurrently; anddetermining the at least one second positional output from the plurality of intermediate second positional outputs by comparing the plurality of intermediate second positional outputs to a respective one of a plurality of depth data.

7. The system as recited in claim 6, wherein the method further comprises:determining the at least one second positional output by comparing an image boundary of respective ones of the plurality of camera image data.

8. The system as recited in any one of the above claims, wherein the method further comprises:determining the estimated position of the target person relative to the mobility device based on a depth data corresponding to the target person.

9. The system as recited in any one of the above claims, wherein the method further comprises:obtaining the at least one LiDAR detection data using a LiDAR module coupled to the mobility device, wherein each of the at least one LiDAR detection data corresponds to at least a lower limb of the target person,obtaining the at least one camera image data using a camera module coupled to the mobility device, wherein each of the at least one camera image data corresponds to at least an upper body image of the target person.

10. The system as recited in claim 9, wherein the LiDAR module defines a 360-degree LiDAR coverage around the mobility device; and the camera module defines a 360-degree field of view around the mobility device.

11. The system as recited in any one of claims 9 and 10, wherein the camera module comprises at least one RGB camera and at least one depth camera.

12. The system as recited in any one of the above claims, wherein the method further comprises performing facial recognition on the at least one second positional output to identify the target person.

13. The system as recited in any one of the above claims, wherein the method further comprises:identifying the target person from a tracking output of the tracker module, wherein the tracking output comprises at least an estimated track of the target person; anddetermining the estimated position of the target person relative to the mobility device based on the estimated track.

14. The system as recited in claim 13, wherein the tracking output comprises a plurality of estimated tracks of a plurality of subjects, wherein the plurality of subjects comprises the target person and at least one non-target person.

15. The system as recited in any one of claims 13 and 14, wherein the method further comprises smoothing the tracking output using a Kalman filter.

16. The system as recited in any one of the above claims, wherein the method further comprises determining a navigation path of the mobility device based on the estimated position of the target person.

17. The system as recited in claim 16, wherein determining the navigation path of the mobility device further comprises:determining an environment context of a surrounding of the mobility device;anddetermining the navigation path based on the environmental context and the estimated position of the target person.

18. The system as recited in any one of claims 16 and 17, wherein determining the navigation path of the mobility device further comprises:determining a behavioural context between a mobility device user and the target person; anddetermining the navigation path based on the behavioural context and the estimated position of the target person.

19. The system as recited in any one of the above claims, wherein the estimated position of the target person relative to the mobility device comprises a 2-dimensional data.

20. An autonomous mobility device, comprising the system as recited in any one of the above claims.