Mobility device, system for and method of side following

Abstract

A system and method for side following. The method comprises determining a plurality of potential paths of a subject to a goal location; determining at least one side goal based on a heading of the subject, wherein each of the at least one side goal is spaced apart from at least one intended path of the subject, the at least one intended path being selected from the plurality of potential paths; determining a computed path of a mobility device based on the at least one side goal, wherein the computed path and the at least one intended path belong to a common homotopy class; and determining a control input for the mobility device using the computed path.

Description

MOBILITY DE VICE, SYSTEM FOR AND METHOD OF SIDE FOLLOWINGCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority to the Singapore application no.10202500459P filed 20 February, 2025, the contents of which are hereby incorporated by reference in their entirety for all purposes.TECHNICAL FIELD

[0002] This application relates generally to the field of autonomous mobility device control, and more particularly, to an autonomous mobility device, a system for side following and a method of side following.BACKGROUND

[0003] An autonomous mobility device, such as an autonomous wheelchair, allows a caregiver or a companion to walk alongside an immobile user (such as an elderly or an injured individual) during movement, which alleviates the need for the caregiver to be positioned behind the wheelchair. In addition, autonomous navigation of the mobility device enables side following of the caregiver, opening up the possibility of side-by-side monitoring and communication between the caregiver and the user. However, various challenges are present in such side following scenarios, such as inaccuracy in determining the path of the mobility device during movement.SUMMARY

[0004] According to an aspect, disclosed herein a system. The system comprises: memory storing instructions; and a processor coupled to the memory and configured to process the stored instructions to implement: a module configured to perform a method of side following The method including: determining a plurality of potential paths of a subject to a goal location; determining at least one side goal based on a heading of the subject, wherein each of the at leastone side goal is spaced apart from at least one intended path of the subject, the at least one intended path being selected from the plurality of potential paths; determining a computed path of a mobility device based on the at least one side goal, wherein the computed path and the at least one intended path belong to a common homotopy class; and determining a control input for the mobility device using the computed path.

[0005] According to another aspect, disclosed herein method of side following, a method of side following. The method includes: determining a plurality of potential paths of a subject to a goal location; determining at least one side goal based on a heading of the subject, wherein each of the at least one side goal is spaced apart from at least one intended path of the subject, the at least one intended path being selected from the plurality of potential paths; determining a computed path of a mobility device based on the at least one side goal, wherein the computed path and the at least one intended path belong to a common homotopy class; and determining a control input for the mobility device using the computed path.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various embodiments of the present disclosure are described below with reference to the following drawings:FIG. 1 is a diagram showing a walking subject, a mobility device, and a system for side following according to embodiments of the present disclosure;FIG. 2A is a schematic diagram showing a system for side following according to various embodiments;FIG. 2B is a data flow diagram for a method of side following of the system of FIG. 2A; FIG. 3 is a data flow diagram for determining a computed path based on a side following scenario according to various embodiments;FIG. 4 schematically illustrates a side following scenario according to various embodiments;FIG. 5 schematically illustrates another side following scenario according to various embodiments,FIG. 6 schematically illustrates another side following scenario according to various embodiments;FIG. 7 schematically illustrates another side following scenario according to various embodiments;FIG. 8 schematically illustrates another side following scenario according to various embodiments;FIG. 9 schematically illustrates another side following scenario according to various embodiments,FIG. 10 schematically illustrates another side following scenario according to various embodiments,FIG. 11 schematically illustrates another side following scenario according to various embodiments;FIG. 12 is a flowchart of a method of side following according to various embodiments; FIG. 13A is a schematic diagram showing another system for side following according to various embodiments;FIG. 13B is a data flow diagram for a method of side following of the system of FIG. 13A; FIG. 14 top and bottom rows show snapshots of wheelchair trajectory when only the heading and the subject's intended path were used to compute the wheelchair goal;FIG. 15 shows a system approach for side following of an exemplary implementation;FIG. 16 illustrates the variables: Heading, Distance and Direction used to compute the cost function in a modified shared control planner;FIGs. 17 and 18 illustrate a two-segment path computation method (FIG. 17) and a three-segment path computation method (FIG. 18);FIG. 19 shows an area / zone of subject motion and its impact on the computation of the side goal sh;FIG. 20 shows simulation scenarios for the experiments: Scenario 1 and 2 tested left turn and right turn; Scenario 3 and 4 tested obstacle avoidance when obstacle in wheelchair path or person path; Scenario 5, 6 and 7 test entering a room via a door;FIG. 21 shows the actual location where the proposed approach was tested with start points and end points, with the narrow turns, glass walls of the location visible. The path taken by the participants are also shown on the map. The real wheelchair used for the trials is also shown with mounted LiDARs and cameras;FIGs. 22A to 23C shows the average rating for survey questions, with preference towards baseline vs proposed method, manual pushing vs side following, in the point of view as the target subject (FIGs. 22A to 22C) and wheelchair user (FIGs. 23A to 23C). Rating scale - 1 (very low) to 10 (very high). Lower ratings are better for (*) marked questions; and FIG. 24 is a schematic diagram of a processor system.DETAILED DESCRIPTION

[0007] The following detailed description is made with reference to the accompanying drawings, showing details and embodiments of the present disclosure for the purposes of illustration. Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments, even if not explicitly described in these other embodiments. Additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments

[0008] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0009] In the context of various embodiments, the term “about” or “approximately” as applied to a numeric value encompasses the exact value and a reasonable variance as generally understood in the relevant technical field, e g., within 10% of the specified value.

[0010] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0011] As used herein, terms “concurrently”, “simultaneously”, “at the same time”, or the like, may refer to events or actions that coincide or overlap within a period of time, regardless of whether the events start at the same time instant, and regardless of whether the events end at the same time instant.

[0012] As used herein, the term “mobility device” may be used interchangeably with the terms “wheelchair”, “transport device”, “moving device”, “transport robot”, “robotic mobility device”, etc. and may generally refer to a device for moving or transporting a user, such as an elderly or an immobile or partially immobile user.

[0013] Conventional approaches towards mobility devices, such as wheelchairs, often include a caregiver (or a companion) pushing the wheelchair between locations. This requires the caregiver to stay behind the wheelchair, thus reducing the ability of the caregiver to monitor and to talk to the wheelchair user.

[0014] As shown in FIG. 1, disclosed herein an autonomous mobility devices which allows the caregiver or companion to move beside the wheelchair user, thus enabling more effective communication and monitoring. In order to perform side following, the autonomous mobility device is configured to perform the challenging task of predicting the intended path of the caregiver, in addition to various environmental, topological or situational scenarios. For example, the autonomous mobility device has to avoid collision with an obstacle in the environment. As another example, the autonomous mobility device has to maintain a relatively close distance with the companion while concurrently avoiding potential collision withsudden / abrupt changes in the walking path of the companion. As yet another example, the autonomous mobility device has to predict the intended path of the companion, which typically is one among various potential paths

[0015] One approach is the tracking of the position and heading of a subject (such as a walking subject) or a companion at every timestep, in addition to interpolating the interpolated position of the subject along the heading As such, the goal location (or intermediate goal location) for the autonomous mobility device (or autonomous wheelchair) may be determined as a side location of the interpolated position. This works well for scenarios with straight line paths without any obstacles. However, the subject often deviates from the heading direction during turning and avoiding obstacles, which often leads to the autonomous wheelchair overshooting or obstructing the human when the human is turning away from or towards it respectively. In addition, the autonomous wheelchair may also be directed towards another side of the obstacle, resulting in a separation between the subject and the autonomous wheelchair, which may end up in a different corridor or room.

[0016] In addressing the above challenge, the trajectory or intended path of the subject may be predicted based on various parameters, such as: the pose and historical heading of the subject, while taking into consideration the environment map. With the trajectory of the subject, the interpolated position of the subject may be obtained and the goal location of the wheelchair may set be as beside the interpolated position. However, due to various prediction algorithm limitations, the predicted trajectory is often inaccurate or not followed exactly by the subject, thus resulting in the wheelchair moving too close to the subject.

[0017] Thus, it is non-trivial for the autonomous wheelchair to maintain side following with a subject without colliding with or obstructing the subject in addition to avoiding taking different paths around obstacle(s).

[0018] According to various embodiments, disclosed herein is an autonomous mobility device for side following a subject. The autonomous mobility device may be provided or integrated with a system for side following. The system may be configured to perform a method of side following a subject.

[0019] The proposed system leverages a shared control algorithm in addressing whether to follow a side goal along a heading of a subject (or a companion) or a side goal along a predicted trajectory of the subject. The shared control algorithm may define a cost function for the abovedescribed path planning or motion planning problem, such that both the various objectives for side following are addressed and optimized.

[0020] In order to maintain the autonomous wheelchair and the subject on a same side of an obstacle, a computed path of the autonomous wheelchair is maintained in a common or same homotopy class as an intended path of the subject. This is in addition to considering various potential paths that the subject may take within the same homotopy class during the computation of the computed path of the autonomous wheelchair. This leads to the outcome of the autonomous wheelchair implicitly giving way to the subject when there exists a possibility of collision with the subject, while concurrently maintaining side following.

[0021] Referring to FIG. 1, in one aspect, the system 100 according to some embodiments of the present disclosure is configurable as an autonomous mobility device 80, such as an autonomous wheelchair 80, for side following a subject 82 such as a walking subject. The proposed system 100 enables side-by-side movement between the subject 82 and the autonomous mobility device 80. According to various implementations, the system 100 may also allow the autonomous wheelchair 80 to side follow a subject on another wheelchair or mobility device. As such, it may be appreciated that the subject 82 need not be a walking subject. In some implementations, the autonomous mobility device 80 may also be a robotic device or a supporting device for providing assistance or support to a subject 82, such as apatient under rehabilitation or physiological treatment. In yet other implementations, the autonomous mobility device 80 may be a companion device for accompanying a subject 82 during movement.

[0022] In various embodiments, the system 100 may comprise a sensor module 200 in signal communication with a processor 900 or a processor system. The sensor module 200 may be configured to provide sensor signals 202 related to a position and / or a heading of the subject 82. The sensor module 200 may comprise a plurality of cameras for determining a heading of the subject 82. In an example, the sensor module 200 may comprise three orbbec RGB-D cameras coupled to the wheelchair 80 to track the subject 82 on the front, left and right of the wheelchair 80. In other examples, the sensor module 200 may comprise one or more stereo cameras or depth cameras. In other embodiments, in addition to the cameras, the sensor module 200 may also include multiple Light Detection and Ranging (LiDARs) sensors. In an example, the sensor module 200 may comprise three 2D LiDARs coupled to the wheelchair 80 for mapping the environment and for obstacle avoidance.

[0023] In various embodiments, the system 100 may also comprise a mobility device control module 300 also in signal communication with the processor 900. The mobility device control module 300 may be configured to receive control input from the processor 900 and control various actuators, such as motors, on the mobility device 80.

[0024] Further referring to FIGs. 2A and 2B, in various embodiments, the system 100 may further comprise a goal module 110. The goal module 110 may be configured to determine a goal location 112 of the subject 82 in an environment. For example, the goal location 112 may be a specific room or a specific location in a room. In other examples, the goal location 112 may be an intermediate goal location on the way to a final goal location, such as a specific room. In yet other examples, the goal location 112 may be one of multiple intermediate goal locations on the way to a final goal location, such as a specific map location. The goal module110 may first obtain a map of the environment based on sensor signals 202 (mapping data) from the sensor module 200, prior to determining the goal location 112 in the environment.

[0025] According to various embodiments, the system 100 may further include a heading module 120 for determining a heading 122 of the subject 82 based on the sensor signals 202 from the sensor module 200. The heading module 120 may be configured to receive images of the subject 82 to perform a vision-based subject tracking algorithm. In various embodiments, the vision-based subject tracking algorithm may include a first machine learning model for subject detection (such as a YOLO v8 model) and a second machine learning model for subject tracking (such as a DeepSORT model). In order to account for multiple cameras, a third machine learning model (such as a person ReID model) may be implemented by using appearance features from both subject ReID and from the DeepSORT model to track the subject. In various implementations, to track the subject in 3D space, each of the tracked subject may be associated with 3D position obtained from the depth image (Z), YOLOv8 subject detection bounding box and camera intrinsic parameters (X and Y).

[0026] In various implementations, to account for subject and wheelchair motion, a Kalman filter may be used for obtaining the position, heading and velocity of all the tracked subjects. When a subject enters the camera frame, the deepSORT model tracker may associate the subject with an unique ID corresponding to appearance features (using the deepSORT model and person ReID), 3D positions, heading and velocity. The unique ID of the target subject may then be side followed by the mobility device 80.

[0027] According to various embodiments, the system 100 may further include a path generation module 130 for determining a plurality of potential paths 132 of the subject 82 to the goal location 112. The plurality of potential paths 132 may include various paths from a location of the subject 82 to the goal location 112. The plurality of potential paths 132 may be non-overlapping paths or piecewise / partially overlapping paths. The plurality of potential paths132 may include Voronoi paths (or paths on a Voronoi diagram), which takes into consideration the map of the environment and obstacle(s) present in the environment. The Voronoi diagram may comprise partitions of the map into cells taking into consideration the obstacles in the map, wherein the Voronoi paths may be boundaries between the cells.

[0028] According to various embodiments, the system 100 may further include an intended path module 135 for determining at least one intended path 134 of the subject 82. The intended path module 135 may select the at least one intended path 134 from the plurality of potential paths 132. In an example, the intended path module 135 may determine or select a single intended path 134 from the plurality of potential paths 132. In other examples, the intended path module 135 may select a plurality of intended paths 134 from the plurality of potential paths 132. The at least one intended path 134 may correspond to the paths in which the likelihood of the subject 82 taking is higher. The intended path module 135 may take into consideration various parameters to determine the at least one intended path 134.

[0029] In various embodiments, the intended path module 135 may select the at least one intended path 134 from the plurality of potential paths 132 based on the heading 122 of the subject 82. In other words, the intended path module 135 may take into consideration the heading 122 of the subject 82 and determine the one or more intended paths 134. In various embodiments, the intended path module 135 may determine the at least one intended path 134 of the subject 82 based on a plurality of historical headings of the subject 82 and a probability of each of the plurality of potential paths 132. In other words, historical headings of the subject 82 may be taken into consideration in addition to a probability of the subject 82 taking each of the plurality of potential paths 132.

[0030] According to various embodiments, the system 100 may further include a side goal module 140 for determining at least one side goal (Sʰ)142 based on the heading 122 of the subject 82. Each of the at least one side goal 142 may correspond to a respective goal locationof the mobility device 80 such that the mobility device 80 may be positioned beside the subject 82 during side following. In various embodiments, each of the at least one side goal 142 is spaced apart from the at least one intended path 134 of the subject 82. Therefore, the side goal module 140 may determine a side goal 142 corresponding to each intended path 134. Each of the at least one side goal 142 may be aligned with mobility device 80 along a mobility device heading 123. The mobility device heading 123 may be parallel to the heading 122 of the subject 82. The at least one side goal 142 may be an interpolated position based on the mobility device heading 123 or the heading 122 of the subject 82.

[0031] According to various embodiments, the system 100 may further include a path computing module 150 for determining a computed path 152 of the mobility device 80 based on the at least one side goal 142. The computed path 152 may correspond to a target path to be taken by the mobility device 80 at a current position and a current timepoint. In various embodiments, the computed path 152 and the at least one intended path 142 may belong to a common homotopy class. By use of a common homotopy class, the computed path 152 and the at least one intended path 142 may be on the same side of an obstacle. In various embodiments, the path computing module 150 may determine the computed path 152 based on one or more of side following scenarios 70 / 75. Further details relating to the side following scenarios 70 / 75 are described in later sections.

[0032] The path computing module 150 may further determine a waypoint (Sp) 153 based on the computed path 152. The path computing module 150 may further determine the waypoint 153 ahead of a respective side goal 142 on the computed path 152, wherein the waypoint 153 may be limited to within a predetermined distance ahead of the side goal 142. In other words, the waypoint 153 may be a distance ahead of the side goal 142 along the computed path 152. The waypoint 153 may be used during motion control to aid in reducing or mitigating jerkiness during motion.

[0033] In an exemplary embodiment, the system 100 may determine a single intended path 134 of the subject 82 and thus generate a single side goal 142 spaced apart from the single intended path 134. As such, the system 100 may generate a computed path 152 based on the single side goal 142, and the computed path 152 may belong to a common homotopy class as the single intended path 134. The system 100 may also determine a waypoint 153 ahead of the side goal 142 on the computed path 152.

[0034] According to various embodiments, the system 100 may further include a shared control module 160 for determining a control input 162 for the mobility device using the computed path 152. The control input 162 may comprise a target linear speed and a target angular speed of the mobility device 80. In various embodiments, the shared control module may be configured for determining the control input 162 for the mobility device using the waypoint 153, the computed path 152, and the heading 122 of the subject 82.

[0035] In various embodiments, the control input 162 may be used by the shared control module 160 to control the mobility device 80, which marks the end of a single iteration. The shared control module 160 may be configured to determine the control input 162 from a plurality of candidate control inputs based on a shared control cost function. Each of the plurality of candidate control inputs may comprise a candidate linear speed (vt) and a candidate angular speed (wt) of the mobility device. The shared control cost function may be a cost function in the form of Cost(vt, wt) = a*Pl(vt, wt) + b*P2(vt, wt) + c*P3(vt, wt), wherein “a”, “b” and “c” correspond to respective weights of each of a plurality of parameters P1 / P2 / P3. The parameters P1 / P2 / P3 may correspond to inputs to the shared control cost function which collectively affects an outcome of the cost function. The shared control module 160 may seek to minimise the cost related to each pair of candidate control inputs (vt, wt) based on the plurality of different parameters. As examples, the plurality of parameters may include: a distancemeasurement to a nearest obstacle, the waypoint 153, the computed path 152, and the heading 122 of the subject 82.

[0036] Referring to FIG. 3, in various embodiments, the path computing module 150 may determine the computed path 152 of the mobility device 80 based on one or more side following scenarios 70 / 75. Based on various side following scenarios 70 / 75, the system 100 may determine a suitable or preferred computed path 152 of the mobility device 80, while taking into consideration various factors. As non-limiting examples, the side following scenario 70 / 75 may be determined based on a relative position between the mobility device 80 / subject 82 and the environment, including obstacle(s) 90. The side following scenario 70 / 75 may also be determined based on the relative position between the mobility device 80 and the subject 82. The side following scenario 70 / 75 may also be determined based on the side goal 142 and the obstacle(s) 90. The side following scenario 70 / 75 may also be determined based on the intended path 134 of the subject 82 and the side goal 142. Determination of the computed path 152 plays a weighted role in the side following role of the mobility device 80.

[0037] FIGs. 4 to 8 schematically illustrate various side following scenarios 70 according to various embodiments of the invention. In various embodiments, responsive to a side following scenario 70, the system 100 may be configured to select or determine the computed path 152 of the mobility device 80 from a plurality of multi-segment paths. For example, the multisegment paths may be a two-segment path or a three-segment path.

[0038] FIG. 4 corresponds to a side following scenario 70A wherein the intended path 134 is a straight path selected from a plurality of potential paths 132 with no obstacle(s) present between the subject 82 and the wheelchair 80. The path computing module 150 may determine a computed path 152 based solely on the side goal 142 As such, the computed path 152 may be a straight path parallel to the intended path 134 of the subject 82. However, this approach may be inconsistent due to the variable / changing heading of the subject 82 during movement.

[0039] Similarly, FIG. 5 corresponds to a side following scenario 70B with no obstacle(s) present between the subject 82 and the wheelchair 80. For the side following scenario 70B, the system 100 may determine an intended path 134 of the subject 82 (the other potential paths are not shown for clarity) and a side goal (Sh) 142 for the mobility device 80. The side goal may be determined based on a mobility device heading 123, which is parallel to the heading 122 of the subject 82. The side goal 142 may be spaced apart (G) from the intended path 134 of the subject 82. In various embodiments, the path computing module 150 may determine a computed path 152 based on a two-segment path. The two-segment path may comprise a first segment 154 and a second segment 156. The first segment 154 may be a segment or a vector between the mobility device 80 and the subject 82. The second segment 156 may correspond to the intended path 134 of the subject 82. The computed path 152 may be determined by smoothing the two-segment path as shown in FIG. 5. The implementation of the two-segment path may reduce the inconsistency due to variable / rapidly changing heading of the subject 82, resulting in a more consistent path planning.

[0040] Further referring to FIG. 6, responsive to a side following scenario 70C wherein the side goal 142 is located inside / interior of an obstacle 90 or at least partially interior of the obstacle 90, the path computing module 150 may determine a computed path 152 based on a two-segment path. The two-segment path may comprise a first segment 154 and a second segment 156. In other words, when the side goal 142 is located or expected to collide with the obstacle 90, a two-segment path is selected to reduce the risk of collision with the obstacle 90. The first segment 154 may be a segment or a vector between the mobility device 80 and the subject 82. The second segment 156 may correspond to the intended path 134 of the subject 82. The computed path 152 may be determined by smoothing the two-segment path (i.e. first segment 154 and second segment 156), as shown in FIG. 5. It may be appreciated that the smoothing of the two-segment path allows a gradual shift in path trajectory, resulting in a non-abrupt or gradual change in motion due to the presence of the obstacle 90. In addition, the two-segment path also allows / enables the mobility device 80 to avoid the obstacle 90 and to remain on the same side of the obstacle as the subject 82

[0041] Further referring to FIG. 7, responsive to a side following scenario 70D, wherein a third segment 158 joining the side goal 142 and the subject 82 crosses an obstacle 90, the path computing module 150 may also determine a computed path 152 based on a two-segment path, with a first segment 154 and a second segment 156. The side following scenario 70D may be indicative of the side goal and the subject 82 being separated by the obstacle 90. As such, a two-segment path may also be utilized in allowing the mobility device 80 to avoid the obstacle 90 as well as to remain on the same side of the obstacle as the subject 82, similar to the side following scenario 70C.

[0042] Referring to FIG. 8, in various embodiments, responsive to a side following scenarios 70E, the path computing module 150 may determine a computed path based on a three-segment path instead of a two-segment path as described in previous embodiments. The side following scenario 70E may correspond to the at least one intended path 134 of the subject 82 turning around an obstacle 90. In addition, the subject 82 may be closer to an obstacle 90 in comparison to the mobility device 80. In the side following scenario 70E, the obstacle 90 and the mobility device 80 may be on opposing sides of the subject 82. In addition, the at least one intended path 134 of the subject 82 may be turning towards the obstacle 90, such as turning around a comer of a wall (obstacle) or into a doorway. In other words, the at least one intended path 134 of the subject 82 and the potential computed path 1521 may be converging such that the mobility device 80 may collide with the subject 82 during motion.

[0043] As shown in FIG 8, the three-segment path may comprise a first segment 154, a second segment 156 and a third segment 158. The first segment 154 may be a segment or a vector between the mobility device 80 and the subject 82. The second segment 156 maycorrespond to the intended path 134 of the subject 82. The third segment 158 may be a vector or line segment joining the side goal 142 and the subject 82.

[0044] According to various embodiments, the computed path 152 may comprise a first computed path 152A and a second computed path 152B. The first computed path 152A may correspond to a path from a current position of the mobility device 80 to the side goal 142. The first computed path 152A may be determined by smoothing a path formed by connecting a position of the mobility device 80, the subject 82 and the side goal 142. The second compute path 152B may be determined by smoothing the second segment 156 and the third segment 158, as shown in FIG. 8. As such, the mobility device 80 may first move via the first computed path 152A followed by the second computed path 152B. This alleviates the risk of the mobility device 80 colliding with the subject 82 during motion, while maintaining the space gap between the at least one intended path 134 and the computed path 152. This also allows a gradual path trajectory around a corner, resulting in a non-abrupt or gradual motion. As such, the path computing module 150 may determine the first computed path 152A with the side goal 142 as a target position, and determine the second computed path 152B by smoothing the second segment 156 and the third segment 158.

[0045] FIGs. 9 to 11 schematically illustrate various side following scenarios 75 according to various embodiments of the invention In various embodiments, responsive to a side following scenario 75, the system 100 may be configured to shift the side goal 142 prior to determining the computed path 152 of the mobility device 80. The side goal module 140 and the path computing module 150 may collectively shift the side goal 142 prior to determining the computed path 152 of the mobility device 80.

[0046] Referring to FIG. 9, in various embodiments, responsive to a side following scenario 75A, wherein the subject 82 is moving ahead around an obstacle 90, the system 100 (such as the side goal module 140 and the path computing module 150) may shift the side goal (Su) 142to an updated side goal 142A (Sh.new). In an example, the side following scenario 75A may correspond to the obstacle 90 being between a current position 82A of the subject 82 and the mobility device 80, as shown in FIG. 9.

[0047] In such a scenario, the updated side goal 142A may be computed spaced apart from a previous position 82B of the subject 82. In other words, the updated side goal 142A may be determined based on the previous position 82B of the subject instead of the current position 82A of the subject 82. The previous position 82B may correspond to one in which a segment or a vector 154 joining the mobility device 80 and the subject 82 is collision-free. In the side following scenario 75A, the updated side goal 142A may be computed based on the previous position 82B of the subject 82 such that the updated side goal 142 is behind the originally computed side goal 142. Based on the updated side goal 142A, the path computing module 150 may determine a computed path 152 and further determine a waypoint (Sp) 153 based on the computed path 152. This allows the mobility device 80 to slow down during movement, avoiding potential collision with the subject 82.

[0048] Referring to FIG. 10, in various embodiments, the subject 82 may define a potential zone of motion 85. The potential zone of motion 85 of the subject 82 may be an area or a zone ahead of a heading 122 of the subject 82. In other embodiments, the potential zone of motion 85 may be a triangular sectional zone ahead of the heading 122 of the subject 82.

[0049] In various embodiments, responsive to a side following scenario 75B, wherein the subject 82 turns to move towards an obstacle 90 such that a side goal (Sh) 142 is within the potential zone of motion 85, the path computing module 150 may shift the side goal (Sh) 142 to an updated side goal 142A (Sh,ncw). In an example as shown in FIG. 10, the mobility device 80 may be adjacent to the obstacle 90 and located between the obstacle 90 and the subject 82 when the subject 82 turns to move towards the obstacle 90. In such a scenario, the system 100 may shift the side goal 142 which is an interpolated position to the updated side goal 142Abased on a current position 82A of the subject 82. As such, the updated side goal 142A may be by the side of or alongside the current position 82A of the subject 82 instead of ahead of the current position 82A of the subject 82. Based on the updated side goal 142A, the path computing module 150 may determine a computed path 152 and further determine a waypoint (Sp) 153 based on the computed path 152. This allows the mobility device 80 to slow down during movement, avoiding potential collision with the subject 82.

[0050] Further referring to FIG. 11, responsive to a side following scenario 75C wherein the subject 82 is walking straight with the mobility device 80 being too close to the subject 82, the path computing module 150 may also shift the side goal (Sh) 142 to an updated side goal 142A (Sh,new). Similarly, the system 100 may shift the side goal 142 to the updated side goal 142A based on a current position of the subject 82 instead of an interpolated position.

[0051] According to another aspect of the present disclosure, disclosed herein is a method of side following a mobile subject. Taking reference to FIGs. 2A and 2B, the method of side following may comprise determining a goal location 112 and obtaining sensor signals 202 using sensors. The method may further comprise determining a heading 122 of a subject 82 using the sensor signals 202 The method may further comprise generating a plurality of potential paths 132 of the subject 82 to the goal location 112, wherein the plurality of potential paths 132 may be Voronoi paths. The method may further comprise determining a side goal 142 based on the heading 122 of the subject 82, and determining an intended path 134 of the subject 82. The intended path 134 may be selected from a plurality of potential paths 132. The intended path 134 is spaced apart from the side goal 142 by a gap spacing The gap spacing may be in a range of 0.2 to 0.7 meters to facilitate communication between the subject 82 and the mobility device user 84

[0052] The method may further comprise determining a computed path 152 of a mobility device 80 based on the side goal 142. The computed path 152 may be determined based onvarious side following scenarios 70 / 75, taking reference to the respective positions of the mobility device 80 and the subject 82 in the environment. In various embodiments, the method may further comprise selecting the computed path 152 of the mobility device 80 from a plurality of multi-segment paths responsive to a side following scenario 70 / 75. The multi-segment paths may be two-segment paths or three-segment paths.

[0053] The computed path 152 may additionally or alternatively be determined based on the heading 122 of the subject 82, the goal location 112 and / or the intended path 134 of the subject 82. The computed path 152 and the intended path 134 may belong to a common homotopy class. The method may further comprise determining a waypoint 153 based on the computed path 152, wherein the waypoint 153 may be ahead of the side goal 142 on the computed path 152.

[0054] The method may further comprise determining a control input 162 based on the computed path 152. The control input 162 may be detennined based on a shared control framework or algorithm, such as a Dynamic Window Approach (DWA) or a shared DWA. The control input 162 may be used to control a mobility device 80. The control input 162 may comprise a linear speed and an angular speed of the mobility device 80.

[0055] The method may be iteratively performed over each control interval. In other words, each iteration of the method may correspond to a respective control interval of the mobility device 80. In various embodiments, the plurality of potential paths 132 may be updated over each iteration. In exemplary embodiments, the plurality of potential paths 132 may be updated through path contraction and path interpolation based on an updated position and / or heading of the mobility device 80, as well as the heading 122 of the subject 82 using the sensor signals 202.

[0056] In various embodiments, the method 700 of side following may be illustrated in a flowchart as shown in FIG. 12. The method 700 comprises in 710, determining a plurality ofpotential paths of a subject to a goal location; in 720, determining at least one side goal based on a heading of the subject, wherein each of the at least one side goal is spaced apart from at least one intended path of the subject, the at least one intended path being selected from the plurality of potential paths; in 730; determining a computed path of a mobility device based on the at least one side goal, wherein the computed path and the at least one intended path belong to a common homotopy class; and in 740, determining a control input for the mobility device using the computed path.

[0057] In various embodiments, the method 700 further comprises: responsive to a side following scenario, selecting the computed path of the mobility device from a plurality of multisegment paths. In various embodiments, the method 700 further comprises: responsive to the side following scenario comprising the at least one intended path of the subject turning around an obstacle, determining the computed path of the mobility device based on a three-segment path. In various embodiments, the method 700 further comprises: responsive to the side following scenario comprising at least one of: the side goal is inside an obstacle and a third segment joining the side goal and the subject crosses an obstacle, determining the computed path of the mobility device based on a two-segment path.

[0058] In various embodiments, the method 700 further comprises: shifting the side goal prior to determining the computed path In various embodiments, the method 700 further comprises: responsive to the side following scenario comprising the subject moving ahead around an obstacle, shifting the side goal to an updated side goal spaced apart from a previous position of the subject. In various embodiments, the method 700 further comprises: responsive to the side following scenario comprising the side goal being within a potential zone of motion of the subject, shifting the side goal to an updated side goal alongside a current position of the subject.

[0059] In various embodiments, the method 700 further comprises: determining the control input from a plurality of candidate control inputs based on a shared control cost function, the shared control cost function comprising a plurality of parameters corresponding to a distance measurement to a nearest obstacle, a waypoint, the computed path, and the heading of the subject.

[0060] In various embodiments, the method 700 further comprises: determining the at least one intended path of the subject based on a plurality of historical headings of the subject and a probability of each of the plurality of potential paths.

[0061] In various embodiments, the method 700 further comprises: determining a plurality of side goals based on the heading of the subject, wherein each of the plurality of side goals is spaced apart from a respective one of a plurality of intended paths, wherein each of the plurality of potential paths corresponds to a respective one of the plurality of intended paths. Further, the method 700 also includes: determining a plurality of potential computed paths of the mobility device based on the plurality of side goals; and selecting the computed path from the plurality of potential computed paths. The method 700 further comprises: determining a waypoint ahead of the side goal on the computed path; and determining the control input for the mobility device using the waypoint, the computed path, and the heading of the subject.

[0062] The method 700 may be iteratively performed such that the plurality of potential paths is updated over each iteration. The method 700 may further comprise: determining the heading of the subject over each iteration based on sensor signals, wherein each iteration corresponds to a respective control interval of the mobility device.

[0063] FIGs. 13A and 13B illustrate an embodiment of a system 100 according to various embodiments In some instances, considering only the most likely path as the intended path may result in less than optimal performance. In various embodiments, all of the plurality of potential paths 132 may be considered during the mobility device side-following or navigation.This allows the mobility device to gather information by slowing down when human intention is ambiguous.

[0064] The system 100 may further comprise a sensor module 200 and a mobility device control module 300 each in signal communication with a processor 900 or a processor system. The sensor module 200 may be configured to provide sensor signals 202 related to a position and / or heading of the subject 82. The mobility device control module 300 may be configured to receive control input from the processor 900 and control various actuators, such as motors, on the mobility device 80. The system 100 may further comprise a goal module 110 and a heading module. The goal module 110 may be configured to determine a goal location 112 of the subject 82 in an environment and the heading module 120 may be configured to determine a heading 122 of the subject 82 based on the sensor signals 202 from the sensor module 200.

[0065] According to various embodiments, the system 100 may further include a path generation module 130 for determining a plurality of potential paths 132 of the subject 82 to the goal location 112. Departing from earlier embodiments, the system 100 may determine each of the plurality of potential paths 132 as a respective intended path 134. As such, the system 100 considers all of the plurality of potential paths 132 as a plurality of intended paths 134, and does not select intended paths from the plurality of potential paths 132. It may be noted that each of the plurality of potential paths 132 may correspond to a respective one of the plurality of intended paths.

[0066] The system 100 may further include a side goal module 140 and a path computing module 150. The side goal module 140 may be configured for determining a plurality of side goals (Sh)142 based on the heading 122 of the subject 82. In various embodiments, each of the plurality of side goals 142 is spaced apart from a respective one of the plurality of potential paths 132 of the subject 82.1

[0067] Each of the plurality of side goals 142 may be aligned with the mobility device 80 along a mobility device heading 123. The mobility device heading 123 may be parallel to the heading 122 of the subject 82. The plurality of side goals 142 may each be an interpolated position based on the mobility device heading 123 or the heading 122 of the subject 82.

[0068] The system 100 may further include a path computing module 150 for determining a plurality of potential computed paths 151 of the mobility device 80 based on the plurality of side goals 142. In addition, the path computing module 150 may select a computed path 152 from the plurality of potential computed paths 151. As such, multiple potential computed paths 151 may be considered by the path computing module 150 prior to determination of the computed path 152. The computed path 152 may be determined based on various criteria such as the different side following scenarios 70 / 75 as described earlier, and / or other parameters such as overtaking speed, ideal relative position / distance between the mobility device 80 and the subj ect 82, etc. In various embodiments, the computed path 152 and each of the plurality of potential computed paths 151 may belong to a common homotopy class. In various embodiments, each of the potential computed paths 151 may belong to a common homotopy class as a respective one of the plurality of potential paths 132. The path computing module 150 may further determine a waypoint (Sp) 153 based on the computed path 152. The path computing module 150 may further determine a waypoint 153 ahead of a respective side goal 142 on the computed path 152, wherein the waypoint 153 may be limited to within a predetermined distance ahead of the side goal 142.

[0069] In an exemplary embodiment, the system 100 may determine multiple intended paths 134 (or potential paths 132) of the subject 82 and thus generate multiple side goals 142 spaced apart from a respective one of the multiple intended paths 134 (or potential paths 132). However, the system 100 may ultimately generate a single computed path 152 based on the various criteria and side following scenarios 70 / 75. The computed path 152 may belong to acommon homotopy class as each of the multiples intended paths 134. The system 100 may also determine a waypoint 153 ahead of the side goal 142 on the computed path 152. Similar to embodiments above, system 100 may further include a shared control module 160 for determining a control input 162 for the mobility device using the computed path 152.

[0070] Exemplary Implementation

[0071] FIGs 14 to 23C illustrates an exemplary implementation of the proposed system and method. FIG. 15 shows a system approach for side following of a subject (such as a human) according to the exemplary implementation. In the initial stages, the intended paths of the subject were enumerated using Voronoi paths to the goal. At every timestep or time interval, the computed path of the wheelchair is determined such that the computed path is in the same homotopy class as the subject’s intended Voronoi path. The wheelchair path waypoint spand human heading-based side goal Sh are then used by a shared control planner to compute optimal linear and angular speed (v*,w*) for the wheelchair.

[0072] The proposed system may comprise various sensors such as 2D LiDARs, 3D LiDARs, RGBD cameras, or a combination of thereof. While conventional methods typically focus on single-camera subject tracking, the proposed system provided multiple-camera tracking for subject re-identification (Re-ID) approaches that recognizes the same subject across multiple camera frames. The proposed system may consider the appearance of the subject and track him / her in 2D visible space.

[0073] The proposed system may track the subject in 3D space as both the mobility device (such as a wheelchair) and the subject are moving, such that the subject may leave and re-enter the camera view at any time. The subject may be tracked in 3D space by extending the Deep SORT algorithm in combination with a Re-ID framework for multiple-camera tracking

[0074] The process of tracking the subject comprises computing the local pose of the subject,p^‘ —xt> yt> wherein xtl, ytlis the 2D position and d is the heading of the subject 82 withrespect to the mobility device frame at time t. Since the camera is moving with the wheelchair, to predict the intended path or intended trajectory of the subject in the environment, a global pose of the subject p^3= x,y, 9?9may be computed with respect to the static world frame based on the localization of the mobility device.

[0075] To side follow the subject, at every timestep t, a prediction on the subject’s path is made. Various possible paths (or potential paths) may be determined using Voronoi paths, linking the paths across time steps through path contraction and interpolation. This linking allows the computation of the probability of the potential paths being the intended path, based on a full history of subj ect’ s poses using principle of MaxEntlOC, as the probability distribution of potential paths in previous timestep may be used as a prior to compute the posterior distribution. The potential path with highest probability may be taken as the intended path Ttof the subject.

[0076] Given the intended path It, the current pose p^at timestep t, the goal location and environment map, the system may compute an optimal linear speed (v’t) and an optimal angular speed (w\) of the mobility device, such that mobility device stays by the side of the subject.

[0077] Planning Algorithm

[0078] Computing a local goal for the mobility device or robot based on the instantaneous heading or intended path of the subject may lead to overshooting or collision of the mobility device with the subject. The implementation employs intention prediction-based shared control framework which takes into consideration or account the instantaneous heading as well as the intended path of the subject.

[0079] 1) Intention Prediction Based Shared Control: Intention prediction-based shared control algorithm enables shared control such that control authority is given to the subject, while concurrently enabling side following by steering the mobility device alongside the intended path of the subject as well as obstacles avoidance. The shared control algorithm may be basedon Dynamic Window Approach (DWA) and Shared-DWA which compute the optimal (v*t,w*t) pair among the candidate pairs by minimising a cost function C. The cost function C is modified such that the subject’s heading and the intended path predicted based on the history of the subject’s headings are both taken into account. In the exemplary implementation, the cost function for each candidate control pair (vt,wt), is defined as:C = 1 - Clearance + Clearance CosttotaiCosttotal ~ Wcmd ' Costcmd + Wpath ' Costpath (1) wherein Clearance measures the distance to nearest obstacle from which collision can be avoided. It is 0 if the collision is imminent and 1 if there is no collision. Costtotal is a weighted sum of two costs with weights wCmd and Wpath determined using a heuristic based on Clearance of all candidate (vt,wt) pairs.Costcmd = wh ■ Heading + wv• Velocity (2)

[0080] Costcmd as defined by Equation 2 measures how well the mobility device follows the current heading of the subject (jx, jy). The Heading measures how well the wheelchair heading with candidate (vt,wt) is aligned with the direction of the vector (jx, jy). Velocity measures how close the linear speed vt of the mobility device relative to the norm of (jx, jy). Weights Wh and wvwere determined empirically and set to 0.3 and 0.7, respectively.CostpaUi = Wdistance ' Distance + Wdirection ■ Direction (3)

[0081] Costpath as defined by Equation 3 measures how well the mobility device is able to traverse the intended path. Distance and Direction measure the euclidean distance and orientation difference between a waypoint spon the intended path and the closest point to that waypoint cp, on the curvilinear trajectory generated after rolling out the candidate pair (vt, wt) (see FIG. 16).

[0082] 2) Side Following using Intention prediction based shared control: Taking into consideration both the heading of the subject and the intended path, the exemplaryimplementation comprises the shared control framework described above for Side following. The intended path of the subject is considered to be the same as the path the mobility device should take to stay by the side of the subject. Thus, to compute Costcmd from the heading of the subject, the subject’s position is interpolated along the heading and compute heading based side goal Sh to the side of the interpolated position (See FIG. 16). A vector joining this point and the position of the mobility device (jx, jy), represents the intended heading of the mobility device with magnitude normalized between 0 and 1. The vector may be normalized by dividing it with a maximum distance allowed between the mobility device and the side goal Sh and capping its norm at 1.

[0083] The Heading and Velocity from the vector (jx, jy) may be computed for shared control. (See FIG 16). For Costpatii, the computed path of the mobility device and the corresponding waypoint spmay be computed for side following. Similarly, Distance and Direction may be computed from waypoint spin a similar manner for shared control (See FIG. 16).

[0084] Computation of wheelchair path using homotopy class

[0085] It was observed that no obstacle(s) will be present between the subject and the mobility device when the computed path of the mobility device to be in the same homotopy class as the subject. Using this observation, the computed path of the mobility device is set to be in the same homotopy class as the intended path of the subject, by smoothing the path joining the mobility device, the subject and the goal location via the intended path of the subject. (See FIG. 17). This may be known as a two-segment path computation method.

[0086] However, in certain scenarios, following this path may lead to the mobility device turning towards the subject. This may be encountered especially when the subject is turning around an obstacle One reason is due to the subject’s intended path being computed as closer to the obstacle while in actual, the subject may take a different path parallel to the obstacle and in the same homotopy class (See FIG. 14). Referring to FIG. 14, in the top row, the wheelchaireither overshoots (middle image) or cannot turn into the door (last image). In the bottom row, the wheelchair successfully enters the door, but goes too close to the subject (middle image) as the subject moves straight before turning instead of following the intended path exactly.

[0087] To address this issue, the side goal Sh may be computed as an intermediate goal of the mobility device, based on the heading of the subject. This may form a first part of the computed path for the mobility device by smoothing the path formed by connecting a position of the mobility device, the subject and the side goal Sh. To get the wheelchair path to the goal location, a second part of the computed path may be computed by smoothing the path connecting the side goal Sh, a position of the subject and the goal location via user’s intended path. (See FIG.18). This may be known as a three-segment path computation method.

[0088] When the side goal Sh is interior or at least partially interior of an obstacle, the system may choose a computed path corresponding to the two-segment path instead of a three-segment path. Similarly, when a line or vector joining the side goal and the subject crosses an obstacle, the system may choose a computed path corresponding to the two-segment path instead of a three-segment path. This may be understood conceptually as the side goal or line connecting the subject to the side goal in collision may correspond to a situation where space around the subject is tight or limited. In such a case, the mobility device will try to go towards the subject, which corresponds to the two-segment path computation method.

[0089] Ideally, based on the computed path, a line or vector joining the mobility device and the subject should always be collision free. However, in some scenarios, if the subject moves too quickly around turns or comers, the mobility device may collide with the corner of the obstacle. In such scenario, the computed path may be determined using the previous position of the subject in which the line / vector joining the subject and the mobility device was collision free.

[0090] Upon determining the computed path, a waypoint spon the computed path is determined. The waypoint spcorresponds to a distance ahead along the computed path, and may be represented by vt * 8, wherein vt denotes the speed of the mobility device at time step t, and 8 is a fixed value which is chosen such that the mobility device will be moving approximately at the same speed as the subject during side following.

[0091] Tn the exemplary implementation, the waypoint spmay be slightly ahead of the side goal Sh (See FIG. 16). This allows the waypoint spto be located on the second part of the computed path, which acts to guide the mobility device towards goal location. When the mobility device is moving ahead of the human, the waypoint spmay be computed far ahead. Therefore, an upper limit may be set on the waypoint spsuch that the waypoint cannot go beyond a fixed distance ahead of the side goal Sh.

[0092] Computation of heading-based side goal

[0093] Along with avoiding obstacles, the mobility device may also be required to avoid collision with the subject. Therefore, the proposed implementation may ensure that the side goal Shand the waypoint spdo not collide with the subject. An area or zone of subject motion within a common homotopy class may be computed or determined, based on the heading and the intended path of the subject (See FIG. 19). If the heading-based side goal Shlies inside (or at least partially interior of) the area / zone of subject motion, the side goal Shtnay be computed or shifted to a side of the subject instead of being an interpolated position. This would bring the side goal Sh outside of the area / zone of subject motion. If the side goal Shis outside the area / zone of subject motion, the waypoint spwill also be outside the area / zone as both parts of the computed path will be outside the area / zone by construction. Explicitly moving the side goal Sh and the waypoint spoutside of the area / zone of subj ect motion also results in the mobility device slowing down and giving way to the subject when there is a possibility of the mobility device obstructing the walking path of the subject.

[0094] Experiments and Results

[0095] To evaluate the proposed system and method, experiments were conducted both in simulation and on an autonomous wheelchair (mobility device). The hypothesis was that with the proposed approach, the autonomous wheelchair would be able to stay by the side of the target subject as much as possible and will give way when needed and does not go uncomfortably close to the subject.

[0096] A, Baseline

[0097] The proposed system and method were compared with a baseline where the side goal Shwas used to control the wheelchair, by setting wCmd to 1 in Eq 1. An alternative way to configure the baseline is to use an optimal path from the wheelchair position to the goal location, and to obtain the waypoint by computing Costpath in Eq 1 for the baseline. However, it was noted that the optimal path used for computing Costpath is independent of the subject’s path, which results in the wheelchair going very close to the subject, especially in tight spaces. Therefore, it was decided to use heading-based side goal Shas the baseline or baseline algorithm.

[0098] B, Simulation experiments

[0099] For quantitative evaluation, automated experiments were conduct in simulation. Below section describes the setup.

[0100] 1) Setup: The simulated environment is a hospital environment, set up in a Gazebo simulator. The mobility device is a differential drive wheelchair robot, and a human dummy was used as a target subject. For the experiments, seven different scenarios (scenarios 1 to 7, see FIG. 20) that a wheelchair might commonly encounter in a real-world hospital setting were identified, including manoeuvres such as making left or right turns, avoiding obstacles in the path, and navigating through doors or narrow gaps.

[0101] For each scenario, the start position and the end goal for the wheelchair and the subject were fixed. The subject and the wheelchair were manually controlled using joysticks such thatthe wheelchair stays by the side of the subject as much as possible and recorded the trajectory of the subject and the wheelchair. Both the recorded trajectories are stored as a sequence of hga>yhya>tuples < ptr,ptr> where ptris the subject’s pose and pt3ris the wheelchair’s pose at time step t. Subscript r denotes the poses are recorded through manual control. This sequence forms the ground truth for each scenario and is used to evaluate the proposed system and method based on the baseline.

[0102] 2) Protocol: For each trial, in a given scenario, the subject was moved along a prerecorded trajectory to ensure consistency across different runs. To introduce variability and test the adaptability of the proposed motion planning algorithm / method, the subject’s walking speed was varied randomly along the trajectory during each trial. To enable this, a noise profile was generated (i.e. using Perlin noise), which provided a smooth and continuous random sequence of values. The noise profile was applied as a speed multiplier to the time between each of the subject’s recorded positions in the trajectory. With this, the subject could move faster or slower at different segments along the path, with speeds varying between 0 and 2 meters per second.

[0103] The simulated wheelchair followed the subject making use of the proposed method as well as the baseline algorithm during each trial. A total of 100 trials were conducted for each scenario and for each algorithm For each trial, the subject trajectory and wheelchair trajectory hwere recorded and stored as a sequence of tuples < ptaa>°,pta° > wherein subscript o denotes the poses were observed during the trial.

[0104] 3) Evaluation Metric: To assess the success of the trial, the success criteria was defined as the wheelchair reaching the goal without colliding with the subject. If the Euclidean distance between the goal and the final wheelchair pose pT° is less than 0.5metres, the wheelchair is hconsidered to have reached the goal. If the Euclidean distance between pty°and pt0for each timestep t is greater than 0.65metres (subject radius is 0.25m and wheelchair radius is 0.4m),the trajectory is considered collision-free. The Success Rate measures the fraction of successful trajectories.

[0105] Further, to evaluate the wheelchair trajectory with respect to subjective criteria like being by the side of subject as much as possible, giving way when needed, etc., the wheelchair trajectory was compared with the ground truth wheelchair trajectory recorded for that scenario.hSince the subject path is fixed for each scenario, if ptg° is the observed subject pose at time t a>3h3a> during the trial, the corresponding recorded wheelchair pose ptrfrom the tuple < ptr,pt3r> hsuch that pt3r< J>= pt3r, denotes the expected wheelchair position at time t based on the manual co control of wheelchair. Euclidean distance (EDt) between the observed wheelchair pose pt5uandpt3rmeasures deviation from the expected pose. Trajectory Euclidean distance (TED) is the average of EDt for all time steps. The metric Average Trajectory Euclidean distance is the average TED across successful trials for a given algorithm and scenario.

[0106] 4) Results and Discussion: Table I shows the results of the experiments.TABLE I SIMULATION EXPERIMENT RESULTSScenario 1 2 3 4 5 6 7 Success rateOur Approach 1,00 1.00 0.79 0.5 0.91 0.97 0.58 Baseline 0.98 0.33 0.06 0.25 0.34 0.44 0.02 Average Trajectory Euclidean DistanceOur Approach 0.34 0.34 0.71 0.65 0.58 0.46 0.840.33Baseline 0.45 0.63 0.53 0.62 0.32 0.54

[0107] The proposed system and method outperformed the baseline for all scenarios in terms of Success Rate. Average Trajectory Euclidean distance is comparable for all scenarios except for scenario 6.

[0108] Collision with the subject is a key reason for lower success rate of the baseline. When the subject made turns towards the wheelchair’s side (as observed in Scenario 2, 4, and 6), the baseline method frequently overshot, resulting in collision, whereas the proposed method showed a significant reduction in such incidents. For Scenario 4, the proposed method still exhibited a relatively high collision rate in comparison to the baseline, primarily due to the subject making late turns that either resulted in incorrect interpretations of the subject’s intention or delayed the correct intentions appearing, giving the motion planning algorithm insufficient time to react.

[0109] While collision measures overshooting of wheelchair to some extent, the simulation experiments do not capture human reaction to wheelchair motion. In fact, in the real world, actual collisions might not readily happen. The subject reaction may be captured through real wheelchair trials.

[0110] Another reason of the lower success rate for both the baseline and the proposed approach may be due to the wheechair getting stuck while turning around corners as observed in scenario 3 and 7. This may be due to the limitation of underlying DWA based motion planning algorithm, and should not happen when the space is wide and not limited.

[0111] For Scenario 7, which was designed to introduce ambiguity with side-by-side doors, the baseline method frequently failed by guiding the wheelchair into the wrong room due to a lack of adaptive response to the subject’s intentions. In contrast, the proposed method was able to more accurately captured these intentions, guiding the wheelchair into the correct room.

[0112] Average trajectory euclidean distance: Given the length and width of the wheelchair is in a range of 50cm, a value in the range of 0.5metres indicates that the trajectory was close to the ground truth trajectory. This is the case for most of the scenarios for both algorithms In scenario 7, this value is higher for the proposed approach as for many successful cases, thewheelchair takes time to turn around the comer into the door. For the baseline, this metric is low for scenario 7 due to a low success rate.

[0113] C Real wheelchair trials

[0114] 1) Setup: In order to obtain subjective feedbacks, real wheelchair trials were conducted with actual subjects. A trial path was defined with various features to test various real-life situations, such as: left turns, right turns, narrow corridors, sharp turns and static obstacles such as glass walls, pillars, etc (see FIG. 21). Other than the trial path, the goal location may be reached by a straight line path between the start location and the goal location. Conventional methods which compute a wheelchair path based only on the final goal location would not be able to take on the trial path.

[0115] 2) Protocol: During each trial, the target subject, whom which the wheelchair would be following, was asked to walk along the defined trial path and the wheelchair, with an actual wheelchair user seated on the wheelchair, would follow the target subject using either the baseline algorithm or the proposed approach. For the experiments, 2 participants were paired for each experiment. Each participant would take turns to be the target subject and the wheelchair user. In each role, they were asked to try each method once. Therefore, each participant performed 4 trials, with 2 as the target subject and 2 as the wheelchair user. For purpose of a fair evaluation, the participants were not informed of the method used for the trials (baseline or the proposed method), and the order of the methods were randomized between trials. In total, 16 healthy subjects participated in the experiments.

[0116] 3) Evaluation Metric: To evaluate perceived safety and reliability of the proposed method from the perspective of both the target subjects and wheelchair users, separate syurvey questions were prepared, as shown in Table II. Each participant was asked to rate each question on a scale of 1 to 10, with 1 corresponding to very low and 10 corresponding to very high during a trial. The participants were also asked to give their preference between 1) manually pushinga wheelchair and making use of the autonomous wheelchair for side following; and between 2) the baseline approach and the proposed approach.TABLE II SURVEY QUESTIONNAIREFOR TARGET AND PATIENT USER S. No Survey Questionnaire for Target UserQI Was the wheelchair by your side whenever it could be?Q2 Did the wheelchair follow you as per your expectation?Q3 Were you consciously looking back to check the wheelchair? Q4 Was the wheelchair able to follow you during turns?Q5 I felt safe while performing the taskQ6 The wheelchair reached the goal successfully with precision Q7 I prefer this control methodS. No Survey Questionnaire for Patient UserQi Did you feel comfortable with the wheelchair motion?Q2 Did the wheelchair follow the person as per your expectation? Q3 I felt safe while performing the taskQ4 The wheelchair reached the goal successfully with precisionQ5 I prefer this control method

[0117] 4) Results and Discussion: The average ratings of each survey question is shown in FIGs. 22A to 23 C where it is shown that the subjects preferred the proposed method in comparison to the baseline method and rated it better on all questions. For all the survey questions, statistical significance t-test were conducted, with Q6, Q7 significant for target subjects, and Q2, Q4 significant for wheelchair users.

[0118] The proposed method vs baseline: Results show that the proposed method is superior in comparison to the baseline, especially in reaching the goal and side following. This could be attributed to the tendency of the wheelchair overshooting when using the baseline method, and the tendency for the wheelchair to be overly close to the subject when subject is making left / right turns, etc. This is particularly observable around turn 3 (see FIG. 21), which leads to the wheelchair being stuck near or behind the pillar, thus separated from the target subject and resulting in several failure trials. The baseline led to a successful trial only when the wheelchair lagged behind the target subject. In contrast, the proposed approach was able to make theseturns and reach the final goal consistently due to the wheelchair being guided by the intended path of the subject.

[0119] The proposed autonomous mobility device and the proposed system for performing a method of side following may be implemented by a processor system 900 as illustrated in the schematic block diagram of FIG. 24. Components of the processing system 900 may be provided within one or more computing device to carry out the functions of the modules or any other modules. One skilled in the art will recognize that the exact configuration or arrangement illustrated in FIG. 24 is provided by way of example only, e.g., each processing system provided may be different and the exact configuration of processing system 900 may vary.

[0120] In embodiments of the present disclosure, the processing system 900 may include a controller 901 and user interface 902. User interface 902 is configured to enable manual interactions between a user and the computing module as required. For this purpose, the processing system 900 includes the input / output components required for the user to enter instructions to provide updates to each of the modules. A person skilled in the art will recognize that components of user interface 902 may vary from embodiment to embodiment but may typically include one or more input devices 935 such as but not limited to a touchscreen, a keyboard, a joystick, a mouse, a microphone, etc. The user interface 902 also includes EEG sensors 933 that can be attached to the user’s head to sense the user’s brain activity. The user interface 902 can also include a media player 940, which can be in the form of one or more playback devices, including but not limited to a display, a speaker, earphones, headsets, etc.

[0121] The controller 901 is configured to be in data communication with the user interface 902 via bus 915. The controller 901 includes memory 920 and processor 905 mounted on a circuit board to process instructions and data, e.g., to perform the method of the present disclosure. The controller 901 includes an operating system 906, an input / output (I / O) interface 930 for communicating with user interface 902, and a communications interface, e.g., a networkcard 950. The network card 950 may, for example, be configured to send data from the controller 901 via a wired or wireless network to other processing devices or to receive data via the wired or wireless network. Wireless networks that may be utilized by the network card 950 include, but are not limited to, Wireless-Fidelity (Wi-Fi), Bluetooth, Near Field Communication (NFC), cellular networks, satellite networks, telecommunication networks, Wide Area Networks (WAN), and etc.

[0122] Memory 920 and operating system 906 are in data communication with central processing unit (CPU) 905 via bus 910. The memory 920 may include both volatile and nonvolatile memory. The memory 920 may include more than one of each type of memory, e.g., Random Access Memory (RAM) 923, Read Only Memory (ROM) 925, and a mass storage device 945. The mass storage device 945 may include one or more solid-state drives (SSDs). One skilled in the art will recognize that the memory described above includes non-transitory computer-readable media and shall be taken to include all computer-readable media except for a transitory, propagating signal. Typically, instructions are stored as program code in the memory but can also be hardwired. Memory 920 may include a kernel and / or programming modules such as a software application that may be stored in either volatile or non-volatile memory.

[0123] Herein, the term “processor” is used to refer generically to any device or component that can process computer-readable instructions, including for example, a microprocessor, microcontroller, programmable logic device, or other computational device. That is, processor 905 may be provided by any suitable logic circuitry for receiving inputs, processing them in accordance with instructions stored in memory, and generating outputs (for example to the memory components or media player 940). In the present disclosure, processor 905 may be a single core or multi-core processor with memory addressable space. In one example, processor905 may be multi-core, comprising—for example—an 8 core CPU. In another example, it could be a cluster of CPU cores operating in parallel to accelerate computations.

[0124] Further, one skilled in the art will recognize that certain functional units in this description have been labelled as modules throughout the specification. The person skilled in the art will also recognize that a module may be implemented as circuits, logic chips or any sort of discrete component. Still further, one skilled in the art will also recognize that a module may be implemented in software which may then be executed by a variety of processor architectures. In embodiments of the disclosure, a module may also comprise computer instructions or executable code that may instruct a computer processor to carry out a sequence of events based on instructions received. In further embodiments, the module may comprise a combination of different types of modules or sub-modules. The choice of the implementation of the modules may be determined by a person skilled in the art and does not limit the scope of the claimed subject matter in any way.

[0125] All examples described herein, whether of apparatus, methods, materials, or products, are presented for the purpose of illustration and to aid understanding, and are not intended to be limiting or exhaustive. Modifications may be made by one of ordinary skill in the art without departing from the scope of the invention as claimed.

Claims

CLAIMS1. A system, comprising:a memory storing instructions; anda processor coupled to the memory and configured to process the stored instructions to implement:a module configured to perform a method of side following, the method including:determining a plurality of potential paths of a subject to a goal location; determining at least one side goal based on a heading of the subject, wherein each of the at least one side goal is spaced apart from at least one intended path of the subject, the at least one intended path being selected from the plurality of potential paths,determining a computed path of a mobility device based on the at least one side goal, wherein the computed path and the at least one intended path belong to a common homotopy class; anddetermining a control input for the mobility device using the computed path.

2. The system as recited in claim 1, wherein the method further comprises: responsive to a side following scenario, selecting the computed path of the mobility device from a plurality of multi-segment paths.

3. The system as recited in claim 2, wherein the method further comprises: responsive to the side following scenario comprising the at least one intended path of the subject turning around an obstacle, determining the computed path of the mobility device based on a three-segment path.

4. The system as recited in claim 3, wherein the three-segment path comprises: a first segment between the mobility device and the subject; a second segment corresponding to the at least one intended path of the subject; and a third segment between the subject and the side goal.

5. The system as recited in claim 2, wherein the method further comprises: responsive to the side following scenario comprising at least one of: the side goal is inside an obstacle and a third segment joining the side goal and the subject crosses an obstacle, determining the computed path of the mobility device based on a two-segment path.

6. The system as recited in claim 5, wherein the two-segment path comprises: a first segment between the mobility device and the subject; and a second segment corresponding to the at least one intended path of the subject.

7. The system as recited in claim 1, wherein the method further comprises: responsive to a side following scenario, shifting the side goal prior to determining the computed path.

8. The system as recited in claim 7, wherein the method further comprises: responsive to the side following scenario comprising the subject moving ahead around an obstacle, shifting the side goal to an updated side goal spaced apart from a previous position of the subject.

9. The system as recited in claim 7, wherein the method further comprises: responsive to the side following scenario comprising the side goal being within a potential zone of motion ofthe subject, shifting the side goal to an updated side goal alongside a current position of the subject.

10. The system as recited in claim 9, wherein the potential zone of motion of the subject is a triangular sectional zone ahead of the heading of the subject.

11. The system as recited in any one of the above claims, wherein the method further comprises: determining the control input from a plurality of candidate control inputs based on a shared control cost function, the shared control cost function comprising a plurality of parameters corresponding to a distance measurement to a nearest obstacle, a waypoint, the computed path, and the heading of the subject.

12. The system as recited in claim 11, wherein each of the plurality of candidate control inputs comprises a candidate linear speed and a candidate angular speed of the mobility device.

13. The system as recited in any one of claims 1 to 12, wherein the method further comprises: determining the at least one intended path of the subject based on a plurality of historical headings of the subject and a probability of each of the plurality of potential paths.

14. The system as recited in any one of claims 1 to 12, wherein the method further comprises: determining a plurality of side goals based on the heading of the subject, wherein each of the plurality of side goals is spaced apart from a respective one of a plurality of intended paths, wherein each of the plurality of potential paths corresponds to a respective one of the plurality of intended paths.

15. The system as recited in claim 14, wherein the method further comprises: determining a plurality of potential computed paths of the mobility device based on the plurality of side goals; and selecting the computed path from the plurality of potential computed paths.

16. The system as recited in any of the above claims, wherein the method further comprises: determining a waypoint ahead of the side goal on the computed path; and determining the control input for the mobility device using the waypoint, the computed path, and the heading of the subject.

17. The system as recited in claim 16, wherein the waypoint is limited to within a predetermined distance ahead of the side goal18. The system as recited in any of the above claims, wherein the method further comprises: determining the heading of the subject based on sensor signals obtained by a sensor module.

19. The system as recited in any of the above claims, wherein the method further comprises: iteratively performing the method, wherein the plurality of potential paths is updated over each iteration.

20. The system as recited in claim 19, determining the heading of the subject over each iteration based on sensor signals.

21. The system as recited in any one of claims 19 and 20, wherein each iteration corresponds to a respective control interval of the mobility device.

22. The system as recited in any of the above claims, wherein the method further comprises: controlling the mobility device using the control input, wherein the control input comprises a linear speed and an angular speed of the mobility device.

23. An autonomous mobility device, comprising the system as recited in any one of the above claims.

24. A method of side following, the method including:determining a plurality of potential paths of a subject to a goal location; determining at least one side goal based on a heading of the subject, wherein each of the at least one side goal is spaced apart from at least one intended path of the subject, the at least one intended path being selected from the plurality of potential paths;determining a computed path of a mobility device based on the at least one side goal, wherein the computed path and the at least one intended path belong to a common homotopy class; anddetermining a control input for the mobility device using the computed path.

Images