XR controlled mobility device

Abstract

In an embodiment, a method is provided to control a mobility device by a mobility control system, where the method includes receiving user input via an extended reality device, wherein the user input comprises an indication of a direction of travel or destination. The method also includes determining a path for the mobility device based on the direction of travel or destination and based on a map of an environment associated with the mobility device and providing, to a motor control device associated with the mobility device, control signals to control movement of the mobility device along the determined path.

Description

P112304W001 1XR CONTROLLED MOBILITY DEVICETECHNICAL FIELD

[0001] The present disclosure relates to a method and system to control a mobility device with the assistance of an Extended Reality device.BACKGROUND

[0002] In indoor and small spaces, a manual wheelchair is easier to control and maneuver than a powered wheelchair. It can also be more convenient to transport the wheelchair, as a manual wheelchair can be folded easily and fit in a storage space of a vehicle such as a car or airplane. Manual wheelchairs also have fewer parts, making them easier to maintain and less likely to break down. Additionally, a powered wheelchair can cost far more than a manual one. For example, entry level powered wheelchairs begin to be priced around $7,000 to $10,000 USD depending on the features and customizations, while manual wheelchairs usually start around $2,000 to $3,000 USD.

[0003] The expenses and disadvantages of the powered wheelchairs can make the manual wheelchair more practical to use in daily life, but manual wheelchairs have drawbacks as well, including the requirement for somebody to push the wheelchair, or the user having enough mobility and strength to self-propel themselves, which can make it difficult to use for people with high level of disability who cannot control their hands (due to e.g. stroke, Cerebral Palsy, Amyotrophic Lateral Sclerosis (ALS), or spinal cord injuries). The World Health Organization estimates that between 250,000 to 500,000 people suffer spinal cord injuries globally every year, many resulting in permanent disabilities including loss of hand control. Moreover, ALS affects around 2.6 per 100,000 people globally and its progressive nature leads to loss of motor functions including hand control. The vast majority of these users are left with power wheelchairs as the only option to use in their daily life.SUMMARY

[0004] In an embodiment, a method to control a mobility device by a mobility control system is provided, where the method includes receiving user input via an extended reality device, wherein the user input comprises an indication of a direction of travel or destination. The method also includes determining a path for the mobility device based on the direction of travel or destination and based on a map of an environment associated with the mobility device andP112304W001 2providing, to a motor control device associated with the mobility device, control signals to control movement of the mobility device along the determined path.

[0005] In an embodiment, the method can also include generating the map of the environment based on a Simultaneous Localization and Mapping (SLAM) process using sensor data received via the extended reality device (XR).

[0006] In an embodiment, the map of the environment is retrieved from a memory.

[0007] In an embodiment, the method further includes determining a location associated with the map of the environment based on location information received from one or more of the extended reality device or a wireless communication network.

[0008] In an embodiment, the user input is received from one or more of an audio sensor, image sensor, motion sensor or controller of the extended reality device.

[0009] In an embodiment, the user input is based on one or more of an audio command, a gaze detection, motion detection, or mechanical input.

[0010] In an embodiment, the method further includes interpreting the user input with a large language model (LLM) to determine the direction of travel or destination indicated.

[0011] In an embodiment, the user input indicates the direction of travel or destination in the map of the environment.

[0012] In embodiment, the mobility control system is executed on one or more computing devices including one or more of the extended reality device, the motor control device, or a cloud or network based computing device.

[0013] In an embodiment, the method further includes distributing computing tasks associated with controlling the mobility device based on a performance parameter comprising one or more of a network load parameter, battery capacity parameter, network availability parameter, or computing resource parameter.

[0014] In an embodiment, a mobility control system to control a mobility device is provided, where the mobility control system includes processing circuitry and memory, the processing circuitry configured to receive user input via an extended reality device, wherein the user input comprises an indication of a direction of travel or destination. The processing circuitry also determines a path for the mobility device based on the direction of travel or destination and based on a map of an environment associated with the mobility device and provides, to a motor control device associated with the mobility device, control signals to control movement of the mobility device along the determined path.

[0015] In an embodiment, the processing circuitry can also generate the map of the environment based on a SLAM process using sensor data received via the extended realityP112304W001 3device.

[0016] In an embodiment, the map of the environment is retrieved from a memory.

[0017] In an embodiment, the processing circuitry can also determine a location associated with the map of the environment based on location information received from one or more of the extended reality device or a wireless communication network.

[0018] In an embodiment, the user input is received from one or more of an audio sensor, image sensor, motion sensor or controller of the extended reality device.

[0019] In an embodiment, the user input is based on one or more of an audio command, a gaze detection, motion detection, or mechanical input.

[0020] In an embodiment, the processing circuitry can also interpret the user input with an LLM to determine the direction of travel or destination indicated.

[0021] In an embodiment, the user input indicates the direction of travel or destination in the map of the environment.

[0022] In embodiment, the mobility control system is executed on one or more computing devices including one or more of the extended reality devices, the motor control device, or a cloud or network based computing device.

[0023] In an embodiment, the processing circuitry can also distribute computing tasks associated with controlling the mobility device based on a performance parameter comprising one or more of a network load parameter, battery capacity parameter, network availability parameter, or computing resource parameter.

[0024] In an embodiment, a non-transitory computer readable medium is provided that includes computer-readable instructions that, when executed by processing circuitry, perform operations including receiving user input via an extended reality device, wherein the user input comprises an indication of a direction of travel or destination. The operations also include determining a path for the mobility device based on the direction of travel or destination and based on a map of an environment associated with the mobility device and providing, to a motor control device associated with the mobility device, control signals to control movement of the mobility device along the determined path.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawing figures incorporated in and forming a part of this specification illustrate several aspects of the disclosure, and together with the description serve to explain the principles of the disclosure.

[0026] Figure 1 shows an example of a mobility control system assisting a mobility device inP112304W001 4navigating to a destination in accordance with some embodiments of the present disclosure;

[0027] Figure 2 shows an example of an Extended Reality device and a mobility control system in accordance with some embodiments of the present disclosure;

[0028] Figure 3 shows a flowchart of a method to control a mobility device by a mobility control system in accordance with some embodiments of the present disclosure;

[0029] Figure 4 shows an example of a communication system in accordance with some embodiments of the present disclosure; and

[0030] Figure 5 shows a wireless device, which may be configured to operate in the communication system of Figure 4.DETAIEED DESCRIPTION

[0031] The embodiments set forth below represent information to enable those skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the following description in light of the accompanying drawing figures, those skilled in the art will understand the concepts of the disclosure and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure.

[0032] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0033] There currently exist certain challenge(s). Extended Reality (XR) devices can now understand the user environment and can control other smart devices within its mapped environment. For example, XR headsets like Magic Leap or Meta Quest can assist in controlling various loT devices such as lights, speakers, or thermostats in a smart home. A manual wheelchair does not normally have any electronics that might enable its control by the use of other devices, such as XR headsets. However, portable compact engines are now an option that can be attached to the wheelchair. These add-on engines may have their own apps, and recently the trend is to provide limited control through smart watches as well.

[0034] Additionally, controlling powered wheelchairs and other mobility devices has gotten a lot of attention recently due to the advancement in sensor technologies. For example, a powered wheelchair has been proposed that can be controlled by the user's brain activity (EEG), head posture, and neck muscle signals (EMG). The core of that system is a standard electric wheelchair. The EEG acquisition device collects brain signals related to motor imagery, which the user generates by imagining movements. Meanwhile, the Neck EMG collection device measuresP112304W001 5muscle activity in the neck to assess levels of fatigue. A depth camera gathers real-time images to track the user’s head posture. Then a Head Posture Estimation Module analyzes the user's head movements, processing this data to generate control commands. Finally, a Virtual Cursor Control Module combines the various signals into movements of a virtual cursor which is used to control the wheelchair. Combining these multiple input modalities could lead to complex calibration and learning processes for the user which can hinder the practicality of the solution for everyday use. Other wheelchairs have been proposed that use augmented reality based steady-state visually evoked potentials (SSVEP) and motor imagery (MI) or other brain-computer interface technologies to control wheelchairs. These examples serve as a rehabilitation tool by guiding and training users to control the power wheelchair through their brain signals. The AR component is only used as a guidance module that displays the stimulation guidance interface to induce the generated SSVEP signals and guide the MI signals. The user wears AR glasses (or looks at a display) that shows flickering visual blocks. Each block flashes at a different frequency, and each frequency corresponds to a specific movement command (e.g., forward, backward, left turn, right turn, and stop). The system relies on the user’s visual focus to initiate the control. When the user looks at one of the flickering blocks, it stimulates a brain response known as SSVEP. Then an EEG acquisition module collects the brain’s response and decodes the SSVEP signal to translate it into movement control instructions to move the wheelchair accordingly. Other systems rely on a combination of Light Detection and Ranging (LiDAR), cameras, onboard computer, and advanced navigation algorithms which increases its complexity and cost of maintenance, and LiDAR sensors are expensive, with a mid-range sensor costing between $1,000 - $10,000 USD. Therefore, using three or more in a wheelchair can result in a very high cost per wheelchair.

[0035] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. The reviewed current techniques described above are expensive to implement and are complicated to use and maintain by the user. Also, they are not possible to use with a manual wheelchair, which is both cheaper and more practical to use in daily life. Disclosed herein is a wheelchair that combines a manual wheelchair with an add-on engine and is controlled using an XR headset. On the headset, the desired environment can be mapped, for example using Simultaneous Localization and Mapping (SLAM) algorithms. The user inputs the destination within the mapped environment (e.g. I want to go to the kitchen). A path planning algorithm (such as A*) is used to navigate through the environment and generate a path from start to destination points. The path is generally represented as a series of waypoints, which are specific coordinates (x, y, z, or possibly orientations like yaw, pitch, roll). These waypoints outline the trajectory that needs to be followed to get to the destination.P112304W001 6

[0036] In an embodiment, the mounted engine can use a Bluetooth connection or other wireless communication protocol, and the path info can be sent in real-time from the headset to the engine. The engine then moves based on the trajectory information, getting the user to the destination point without the need to use their hand in order to propel the wheelchair.

[0037] The mobility control system described herein combines the advantages of using a manual wheelchair, in terms of compact lightweight, low cost and practicality in daily life, with the mobility of battery powered wheelchair for disabled users who have limited to no hand control. The solution includes a manual wheelchair, a detachable engine, and an XR headset.

[0038] An XR headset is used to map the user’s environment. Combined with user’s input, the system determines starting and destination points and an obstacles free trajectory.

[0039] The user input can be voice, text, or eye gaze, e.g. look to the left to go to the left. In other embodiments, the user input could include hand gestures and / or other movements, such as blinking and nodding.

[0040] A large language model (LLM) is used to extract meaningful instructions from the user’s input, (e.g. start moving left towards the room door then forward towards the kitchen). This operation can be learnt based on the environment too. The mounted engine is then directed to move the wheelchair based on the information sent from the headset.

[0041] While reference is made herein to wheelchairs, it should be appreciated that the principles described herein of using a mobility control system to control a mobility device with the assistance of an XR device are equally applicable to other embodiments, such as mobilized scooters, automobiles, and other motorized devices that can transport people.

[0042] Certain embodiments may provide one or more of the following technical advantage(s):• The solution operates on a manual, not powered, wheelchair, which is more practical in daily life.• Much cheaper to implement than adding LiDAR sensors, cameras, and an on-board computer to control the wheelchair.• Easier maintenance due to the simplicity of the system.• Opens the manual wheelchair option for users with limited to no hand control.

[0043] Figure 1 shows an example of a mobility control system 106 assisting a mobility device 104 in navigating to a destination in accordance with some embodiments of the present disclosure.

[0044] Descriptions of how the mobility control system 106 may be implemented will be described in more detail in Figure 2, while Figure 1 broadly describes the functionality performed by the mobility control system 106.P112304W001 7

[0045] As described above, while mobility device 104 is depicted as a user in a wheelchair, other implementations such as mobilized scooters, automobiles, and other motorized devices that can transport people are applicable in other embodiments.

[0046] Further, the XR device 204 is depicted in Figure 1 as a virtual reality headset, but other implementations are possible, such as any device that includes a display screen with one or more image, audio, motion sensors, or a controller that includes one or more means of inputting user input manually (e.g., a joystick, keyboard, mouse, or other mechanical input device). The XR device 204 can be wearable, be carried by the user, or be mounted or attached to the mobility device 104. The mobility control system 106 can be implemented or executed by one or more computing devices, including the XR device 204, a motor control device on the mobility device 104, or in the cloud or network-based computing device.

[0047] The mobility control system 106 can determine a position or location of the mobility device 104 within the environment 102 and then can assist the user in determining a path 112 to a destination 108, and also provide signaling to a motor control device of the mobility device 104 to propel the mobility device 104 to the destination. The path 112 can take into account any obstacles 110 that are present in the environment 102.

[0048] The mobility control system 106 can have a stored map of the environment 102 and retrieve the map based on determining that the mobility device 104 is within the environment 102 based on recognizing the obstacles and other features of the area. The mobility control system 106 can also use location information, such as Global Positioning System (GPS) coordinates from a GPS system, or network location information in order to determine a location of the mobility device 104 and correlate the coordinates or location with the stored map of the environment 102. In other embodiments, the mobility control system 106 can use SLAM to generate a map of the environment 102 using the image sensors on the XR device 204.

[0049] In an embodiment, if the map is known, the user can use input such as audio command, a gaze detection, motion detection, or mechanical input to specify the destination 108 even if it is not in a line of sight to the XR device 204. In other embodiments, where SLAM is being performed, and the general map of the environment 102 is not already known, the user can provide input to direct the mobility device 104 to somewhere within a line of sight. In yet another embodiment, the user can provide input instructing the mobility control system 106 to find a path around an obstacle, even if the map is not already stored.

[0050] In an embodiment, the XR device 204 can include an environmental sensing system (e.g. a camera system, other environmental sensor) having one or more sensors configured to capture information characterizing a surrounding environment. This user input device (a “non-P112304W001 8manual device” such as an XR glasses, gaze detector, a camera, audio sensor for spoken commands) can be configured to receive input (the input can indicate a destination, an instruction to deviate from a path, etc.) from a user of a manual wheelchair.

[0051] The mobility control system 106 can be configured to control a detachable motor connectable to the mobility device 104 (for practical purposes it is significant that the motor is designed for connection to and disconnection from a compact wheelchair, not a typical fully powered chair) to propel the mobility device 104 along a path in the surrounding environment based on captured information characterizing the surrounding environment and based on received input via the user input device.

[0052] Figure 2 shows an example of the XR device 204 and a mobility control system 106 in accordance with some embodiments of the present disclosure. The mobility control system 106 can direct a motor control device 202, which can be a separate detachable motor unit that can be attached to a wheelchair or other manually controlled vehicle.

[0053] The mobility control system 106 can be executed on any of the XR device 204, the motor control device 202 that is attachable to the mobility device 104 (e.g., a wheelchair, an automobile, or other device, etc.), or in a cloud or network 216. The mobility control system 106 may also be implemented on any other computing device communicably coupled to the XR device 204 and / or the motor control device 202. In some embodiments, the mobility control system 106 can be implemented at a single location, and in other embodiments, the mobility control system 106 can be implemented at more than one location, with one or more functions being performed at different locations. For example, in a non-limiting example, the XR device 204 can implement a portion of the mobility control system 106 that performs SLAM or performs some processing of the user input, while other portions of the processing of the user input (e.g., computationally intensive portions) or path generation can be performed in the cloud or network 216.

[0054] In various embodiments, the mobility control system 106 can distribute computing tasks between the different instantiations of the mobility control system 106 based on a performance parameter such as a network load parameters, battery capacity parameter of the XR device 204, network availability parameter (e.g., wireless communication service to the XR device 204 and / or motor control device 202) or computing resource parameter (e.g., how computationally intensive the tasks are compared to the computing resources available at the XR device 204, the or motor control device 202.

[0055] The mobility control system 106 can first determine a location of the mobility device 104 within an environment. To do so, the mobility control system 106 can either retrieve location information from the XR device 204, mobility device 104, or receive location information from aP112304W001 9wireless communication system. The location information retrieved from the XR device 204, mobility device 104, or wireless communication system can be in the form of GPS coordinates, or other coordinate system and can be used to determine a location of the mobility device 104. The location and / or location information can be used by the mobility control system 106 to see if there are any matches with any stored map of the environment 102. The location information could also be correlated to a map identifier of a stored map of the environment 102 wherein the environment 102 is the area that is around or near a location identified by the location information.

[0056] Once the location is known, the mobility control system 106 can determine that there is a stored map associated with the environment 102 and retrieve the map from a memory - either in the network, or any of the devices on which the mobility control system 106 is operating.

[0057] In other embodiments, the mobility control system 106 can utilize image sensors 210 (e.g., cameras, range- finding sensors, etc.) in order to perform SLAM. SLAM is a computational process used to enable a robot, drone, or autonomous system to build a map of an unknown environment while simultaneously determining its position within that map. The process begins with the system capturing data from the image sensors 210 which collect information about the surrounding environment. Algorithms then process this data to identify key landmarks, which are distinctive features or objects in the environment that can serve as reference points. By comparing these landmarks across consecutive sensor readings, the mobility control system 106 can estimate the position and orientation of the mobility device 104 within its environment.

[0058] The mobility control system 106 can then, based on user input, determine the destination (e.g., destination 108) to which the user wants to direct the mobility device 104, and also the path 112 to reach the destination.

[0059] The user input can be received via one or more of audio sensors 208 (e.g., in the form of a spoken request or command from the user), from image sensors 210 (e.g., in the form of pointing, gaze detection, etc.,), motion sensors 220 (e.g., gestures, blinking, nodding, etc.) or from one or more controllers 212 (e.g., keyboard, joystick, handheld controller, etc.). The user input can specify either a destination or give directions for a path. For example, if the map is previously unknown, the user can give directions to a waypoint that is within line of sight to one or more of the image sensors 210. One or more of the image sensors 210 and / or electrooculographic (EOG) sensors can also track movement of the eyes in order to extract information about where the user intends to direct the mobility device 104 and correlate the gaze direction to a destination. The eye gaze can also be used to execute the user’s command to start moving the wheelchair. It can also be used as a safety measure to stop the engine when, for example, the user looks at their lap or blinks a predefined number of times.P112304W001 10

[0060] In an embodiment, to determine a destination or path from user input received from the audio sensors 208, an LLM model can be used. An LLM example could be the new multimodal Llama (3.2) that has vision models capability that can be run on-device. The user input can be prompted as text or voice instructions, combined with eye gaze tracking to guide where the user’s next move is. Moreover, combining where the user is looking, with the LLM instructions extracted from other methods of input (text, voice), can add an extra layer of safety that prevents the engine from driving into obstacles, especially in a dynamic environment.

[0061] When planning the path 112 to the destination 108, the mobility control system 106 can take into account any obstacles 110 and avoid them when planning the movements of the mobility device 104. The mobility control system 106 can also preferentially direct the mobility device along smooth and / or flat surfaces, roadways, and avoid any potential collisions with pedestrians, vehicles, or other moving objects. Once the path is planned, the mobility control system 106 can provide control signals to the drive mechanism 218 of the motor control device 202.

[0062] Figure 3 shows a flowchart of a method to control a mobility device by a mobility control system in accordance with some embodiments of the present disclosure.

[0063] The operations 302 of receiving user input and 304 of localization and mapping can happen independent of each other and at the same time or different times.

[0064] At operation 302, the mobile control system can receive the user input via an extended reality device (e.g., XR device 204), wherein the user input comprises an indication of a direction of travel or destination. The user input can be received via one or more of an audio sensor, image sensor, motion sensor or controller of the extended reality device and can include audio command, a gaze detection, motion detection, or mechanical input.

[0065] At operation 304, the mobile control system 106 can receive location information 308 and retrieve a map for the area based on the map. If the location is unknown, or there is no map available for the location, the mobile control system 106 can perform SLAM at operation 306 to build a map for the location based on image data and other sensor data.

[0066] At operation 310, the mobile control system can use an LLM agent to interpret the user input (e.g., from the audio command, text entered via a keyboard, gaze detection) in order determine the direction of travel or destination (108) indicated based on the map. Once the destination is known or determined, the mobile control system 106 can, at operation 312 plan a path to the destination, and at operation 314, provide a control signal to the motor control device 202 to direct the mobility device 104.

[0067] Figure 4 shows an example of a communication system 400 in accordance with someP112304W001 11embodiments.

[0068] In the example, the communication system 400 includes a telecommunications network 402 that includes an access network 404, such as a radio access network (RAN), and a core network 406. The access network 404 includes one or more access network nodes or base stations of various types, access network nodes 410A and 410B are depicted (which may be collectively referred to as network nodes 410), or any other similar 3rdGeneration Partnership Project (3GPP) access nodes or non-3GPP access points (APs). Some embodiments of the access network 404 may include more than one access network technology. The network nodes 410 of access network 404 facilitate direct or indirect connection to the mobility control system 106 of either the motor control device 202 or the extended reality device 204. The core network 406 may also include an instantiation of the mobility control system 106 or an instantiation of the SLAM / object detection network that facilitates navigation by the mobility control system 106.

[0069] Moreover, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunications network 402 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a network node in the telecommunications network 402 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other network nodes to implement one or more functionalities of any network node in the telecommunications network 402 including one or more access network nodes 410.

[0070] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU-CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). An ORAN network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an Al, Fl, Wl, El, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN network node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an O-2 interface definedP112304W001 12by the O-RAN Alliance or comparable technologies.

[0071] The network nodes 410 facilitate direct or indirect connection of the motor control device 202 or the XR device 204 to the core network 406 over one or more wireless connections. Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 400 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 400 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0072] As a whole, the communication system 400 of Figure 4 enables connectivity between the motor control device 202, XR device 204, network nodes, and core network. In that sense, the communication system 400 may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (Wi-Fi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (Wi-Max), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, Li-Fi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox. Moreover, the communication system 400 may be configured to support multiple different standards, protocols, or other rule sets, with individual components supporting all of the relevant rule sets or with different components or sub-systems within the communication system 400 supporting different standards, protocols, or rule sets.

[0073] As one example, in certain embodiments, access network 404 may contain some access network nodes 410 that support 3GPP radio access technologies (RAT), such as LTE or NR, while other access network nodes 410 support (or the same access network nodes 410 additionally support) non-3GPP RATs, such as Wi-Fi or a proprietary RAT. As another example, telecommunications network 402 may support multiple generations of related communication standards (e.g., 4G and 5G 3GPP communication standards) and, as a result, may include an access network and / or a core network that supports multiple different standard generations or may includeP112304W001 13multiple access networks and / or multiple core networks with individual networks supporting different standard generations.

[0074] Telecommunications network 402 may support network slicing to provide different logical networks to different devices that are connected to the telecommunications network 402. For example, the telecommunications network 402 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive loT services to yet further UEs.

[0075] Figure 5 shows a mobility control system 106, which may be configured to operate in communication system 400 of Figure 4 or on XR device 204 or motor control device 202.

[0076] In particular embodiments, mobility control system 106 includes processing circuitry 502 that is operatively coupled via a bus 504 to an input / output interface 506, a power source 508, a memory 510, a communication interface 512, and / or any other component, or any combination thereof. Certain embodiments of mobility control system 106 may include all or a subset of the components shown in Figure 5. The level of integration between the components may vary from one embodiment of mobility control system 106 to another. In general, in a particular embodiment of mobility control system 106, processing circuitry 502, input / output interface 506, power source 508, memory 510, and communication interface 512 may, in whole or in part, represent or include physical components common to or shared by one or more of the other elements of mobility control system 106. Further, certain embodiments of mobility control system 106 may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0077] The processing circuitry 502 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 510. The processing circuitry 502 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 502 may include multiple central processing units (CPUs).

[0078] In the example, the input / output interface 506 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor,P112304W001 14a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into mobility control system 106. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presencesensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0079] In some embodiments, the power source 508 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used to supply power to circuitry or to charge an associated battery. The power source 508 may further include power circuitry for delivering power from the power source 508 itself, and / or an external power source, to the various parts of mobility control system 106 via input circuitry or an interface such as an electrical power cable. Power source 508 may perform any formatting, converting, or other modification to make accessible power suitable for the respective components of the mobility control system 106 to which power is supplied.

[0080] The memory 510 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 510 includes one or more programs 514, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 516. The memory 510 may store, for use by mobility control system 106, any of a variety of various operating systems or combinations of operating systems.

[0081] The memory 510 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) includingP112304W001 15one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 510 may allow mobility control system 106 to access instructions, programs, and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 510, which may be or comprise a device-readable storage medium.

[0082] The processing circuitry 502 may be configured to communicate with an access network or other network via or using the communication interface 512. The communication interface 512 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 522. The communication interface 512 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., a wireless device or a network node in an access network, or the XR device 204 or the motor control device 202). Each transceiver may include a transmitter 518 and / or a receiver 520 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 518 and receiver 520 may be coupled to one or more antennas (e.g., antenna 522) and may share circuit components, software, or firmware, or alternatively be implemented separately.

[0083] In the illustrated embodiment, communication functions of the communication interface 512 may include cellular communication, Wi-Fi communication (e.g., according to an IEEE 802.11 family standard), LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0084] In particular embodiments, mobility control system 106 may provide an output of data captured via a sensor, through its communication interface 512, via a wireless connection to a network node, and / or in any appropriate manner. Data captured by sensors of a mobility control system 106 can be communicated through a wireless connection to a network node via anotherP112304W001 16mobility control system 106. In particular embodiments, such output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0085] As another example, mobility control system 106 comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, mobility control system 106 may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0086] Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions, and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0087] In certain embodiments, some or all the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionalities may beP112304W001 17provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0088] Those skilled in the art will recognize improvements and modifications to the embodiments of the present disclosure. All such improvements and modifications are considered within the scope of the concepts disclosed herein.

Claims

P112304W001 18CLAIMS1. A method to control a mobility device (104) by a mobility control system (106), the method comprising:receiving (302) user input via an extended reality device (204), wherein the user input comprises an indication of a direction of travel or destination (108);determining (312) a path (112) for the mobility device (104) based on the direction of travel or destination (108) and based on a map of an environment (102) associated with the mobility device (104); andproviding (314), to a motor control device (202) associated with the mobility device (104), control signals to control movement of the mobility device (104) along the determined path (112).

2. The method of claim 1, further comprising:generating (306) the map of the environment (102) based on a simultaneous localization and mapping, SLAM, process using sensor data received via the extended reality device (204).

3. The method of claim 1, wherein the map of the environment (102) is retrieved from a memory (510).

4. The method of claim 3, further comprising:determining (308) a location associated with the map of the environment (102) based on location information received from one or more of the extended reality device (204) or a wireless communication network.

5. The method of any of claims 1 to 4, wherein the user input is received from one or more of an audio sensor (208), image sensor (210), motion sensor (220) or controller (212) of the extended reality device (204).

6. The method of any of claims 1 to 5, wherein the user input is based on one or more of an audio command, a gaze detection, motion detection, or mechanical input.

7. The method of claim 6, further comprising:interpreting (310) the user input with a large language model, LLM, to determine theP112304W001 19direction of travel or destination (108) indicated.

8. The method of any of claims 1 to 7, wherein the user input indicates the direction of travel or destination (108) in the map of the environment (102).

9. The method of any of claims 1 to 8, wherein the mobility control system (106) is executed on one or more computing devices including one or more of the extended reality device (204), the motor control device (202), or a cloud or network based computing device.

10. The method of claim 9, further comprising:distributing (316) computing tasks associated with controlling the mobility device (104) based on a performance parameter comprising one or more of a network load parameter, battery capacity parameter, network availability parameter, or computing resource parameter.

11. A mobility control system (106) to control a mobility device (104), the mobility control system (106) comprising processing circuitry (502) and memory (510), the processing circuitry (502) configured to:receive (302) user input via an extended reality device (204), wherein the user input comprises an indication of a direction of travel or destination (108);determine (312) a path (112) for the mobility device (104) based on the direction of travel or destination (108) and based on a map of an environment (102) associated with the mobility device (104); andprovide (314), to a motor control device (202) associated with the mobility device (104), control signals to control movement of the mobility device (104) along the determined path (112).

12. The mobility control system (106) of claim 11, wherein the processing circuitry (502) is further configured to:generate (306) the map of the environment (102) based on a simultaneous localization and mapping, SLAM, process using sensor data received via the extended reality device (204).

13. The mobility control system (106) of claim 11, wherein the map of the environment (102) is retrieved from the memory (510).P112304W001 2014. The mobility control system (106) of claim 13, wherein the processing circuitry (502) is further configured to:determine (308) a location associated with the map of the environment (102) based on location information received from one or more of the extended reality device (204) or a wireless communication network.

15. The mobility control system (106) of any of claims 11 to 14, wherein the user input is received from one or more of an audio sensor (208), image sensor (210), motion sensor (220) or controller (212) of the extended reality device (204).

16. The mobility control system (106) of any of claims 11 to 15, wherein the user input is based on one or more of an audio command, a gaze detection, motion detection, or mechanical input.

17. The mobility control system (106) of claim 16, wherein the processing circuitry (502) is further configured to:interpret (310) the user input with a large language model, LLM, to determine the direction of travel or destination (108) indicated.

18. The mobility control system (106) of any of claims 11 to 17, wherein the user input indicates the direction of travel or destination (108) in the map of the environment (102).

19. The mobility control system (106) of any of claims 11 to 18, wherein the mobile control system (106) is executed on one or more computing devices including one or more of the extended reality device (204), the motor control device (202), or a cloud or network based computing device.

20. The mobility control system (106) of claim 19, wherein the processing circuitry (502) is further configured to:distribute (316) computing tasks associated with controlling the mobility device (104) based on a performance parameter comprising one or more of a network load parameter, battery capacity parameter, network availability parameter, or computing resource parameter.

21. A non-transitory computer readable medium comprising computer-readable instructionsP112304W001 21that, when executed by processing circuitry (502), perform operations, comprising:receiving (302) user input via an extended reality device (204), wherein the user input comprises an indication of a direction of travel or destination (108);determining (312) a path (112) for a mobility device (104) based on the direction of travel or destination (108) and based on a map of an environment (102) associated with the mobility device (104); andproviding (314), to a motor control device (202) associated with the mobility device (104), control signals to control movement of the mobility device (104) along the determined path (112).

Images