Server apparatus, positioning system and method for estimating relative position of an autonomous aerial device with respect to a ground device

Abstract

Aspects concern a server apparatus or system for estimating a relative position of an autonomous aerial device with respect to a ground device, the server apparatus comprising at least one processor configured to: obtain, from an inertial measurement unit (IMU) of the autonomous aerial device, one or more measurements of the autonomous aerial device; obtain, from an inertial measurement unit (IMU) of the ground device, one or more measurements of the ground device; generate a plurality of rotation matrices based on the one or more measurements of the autonomous aerial device and the one or more measurements of the ground device; detect, using a marker-based detection method, image data associated with one or more visual markers of the ground device, and estimate a positional dataset of the autonomous aerial device based on the image data; and estimate the relative position based on the plurality of rotation matrices, and the positional dataset of the autonomous aerial device.

Description

SERVER APPARATUS, POSITIONING SYSTEM AND METHOD FOR ESTIMATING RELATIVE POSITION OF AN AUTONOMOUS AERIAL DEVICE WITH RESPECT TO A GROUND DEVICECROSS-REFERENCE TO RELATED APPLICATION

[0001] The application claims the benefit of priority of Singapore patent application No.10202403374U, filed 30 October 2024, the content of it being hereby incorporated by reference in its entirety for all purposes.TECHNICAL FIELD

[0002] The disclosure relates to a device, system and method of estimating a relative position of an autonomous aerial device with respect to a ground device, such as a robot. In particular, the disclosure may be applied, but is not limited to, a tethered drone and ground robot navigation system.BACKGROUND

[0003] The following discussion of the background is intended to facilitate an understanding of the present disclosure only. It should be appreciated that the discussion is not an acknowledgement or admission that any of the material referred to was published, known or is part of the common general knowledge of the person skilled in the art in any jurisdiction as of the priority date of the disclosure.

[0004] A multi-robot system may be deployed to work at different locations simultaneously, handling multiple tasks at once to accomplish an objective or a mission. The relative success in such mission hinges on accurate localization of each robot with respect to the environment and with respect to other robots within the multi-robot system. Current multi-robot system may include a tethered drone and a ground robot.

[0005] There are many ways to obtain relative localization or position between a tethered drone and a ground robot. A relatively straightforward method is to use a global navigation satellite system (GNSS) to obtain the global pose of each robot and calculate the relative pose therebetween. However, the GNSS method suffers from inaccuracy, especially in urban environments like cities, where tall buildings surround the robots. Moreover, the use of GNSS may not be suitable for security-critical applications because the GNSS signals may be spoofedor jammed. In some instances, the unavailability of GNSS signals in indoor missions restricts the capabilities of the robots only to outdoors.

[0006] Another method for relative localization is the use of motion capture systems. Such motion capture systems can provide relatively accurate pose estimation of robots by high-frequency tracking cameras. However, they are typically not practical for real-world deployment because they require substantial installation and calibration effort. Such motion capture systems are typically used in labs for accuracy-critical robotics research, e.g., swift maneuvers of drones.

[0007] Odometry-based methods constitute another branch for relative localization of robots. These methods include Light Detection and Ranging LiDAR-InertiaL or LiDAR-Odometry and Visual-Inertial- or Visual-Odometry. These methods typically estimate how far each robot has travelled from their starting position. If the starting poses of both robots are known precisely, these methods can provide the relative pose between the robots on-the-fly. Odometry-based methods do not rely on any external infrastructure, rendering them suitable for real-world missions. However, their critical drawback is that they tend to accumulate errors over time. They also yield inaccurate results when visual features in the environment are insufficient or when the robot performs abrupt motions, e.g., a drone recovering from a wind gust.

[0008] Another way to estimate the relative localization between a tethered drone and a ground robot is to leverage on the physical connection between the robots, i.e., tether cable. If the tether cable is taut during an operation, the relative position between the robots can be calculated by measuring the cable length and the azimuth & elevation angles. However, this method yields inaccurate estimation when the cable is not perfectly taut. In real-world missions, the tether cable is almost never taut, especially when the distance between the robots increases. Even with a proper winch design, the cable tends to sag because the torque by the winch and the pulling force by the drone become insufficient to stretch a very long tether cable.

[0009] In view of at least the aforementioned problems or inadequacies, there exists a need to provide a technical solution to enable or provide a more robust system, device or method for obtaining accurate positional measurements and / or estimate the relative position of drone with respect to ground robots.SUMMARY

[0010] A technical solution may be provided in the form of accurate relative localization of robots which may be vital for proper coordination in multi-robot missions. An approach for relative localization of an autonomous aerial device (e.g., a tethered drone) & a ground device (e.g., ground robot platform) in the form of a system which may be referred to as Tethered Drone & Ground Robot Tandem Navigation (TeDRO). The relative positioning of an autonomous aerial device with respect to the ground robot may be performed without any external infrastructure, e.g., Global Navigation Satellite System (GNSS). This enables the drone and robot to operate in unstructured environments or in security-critical missions without worrying about the quality or authenticity of GNSS signals.

[0011] In some embodiments, the autonomous aerial device may be equipped with an image capturing device arranged in a particular orientation to detect the one or more visual markers, for example, a downward-facing camera. Such an arrangement may be used to facilitate the detection of ground robot in aerial images and estimates its relative pose on-the-fly. The estimated relative pose may be compared with measurements obtained from Inertial Measurement Unit (IMU) data of the autonomous aerial device, and may be dynamically corrected based on the compared IMU data. The proposed approach has also enabled real-time operability of the ground device and the autonomous aerial device in both indoors and outdoors.

[0012] According to an aspect of the present disclosure there is provided a server apparatus for estimating a relative position of an autonomous aerial device with respect to a ground device, the server apparatus comprising at least one processor configured to: obtain, from an inertial measurement unit (IMU) of the autonomous aerial device, one or more measurements of the autonomous aerial device; obtain, from an IMU of the ground device, one or more measurements of the ground device; generate a plurality of rotation matrices based on the one or more measurements of the autonomous aerial device and the one or more measurements of the ground device; detect, using a marker-based detection method, image data associated with one or more visual markers of the ground device, and estimate a positional dataset of the autonomous aerial device based on the image data; and estimate the relative position based on the plurality of rotation matrices and the positional dataset of the autonomous aerial device. In some embodiments, the at least one processor is further configured to estimate a heading, in addition to the relative position, based on the plurality of rotation matrices and the positionaldataset of the autonomous aerial device. The heading and the relative position may form a relative pose of the autonomous aerial device with respect to the ground device.

[0013] In some embodiments, the plurality of rotation matrices comprises a first rotation matrix and second rotation matrix, wherein the first rotation matrix is generated based on the one or more measurements of the IMU of the autonomous aerial device, and the second rotation matrix is generated based on the one or more measurements of the IMU of the ground device.

[0014] In some embodiments, the processor is configured to generate a shared coordinate system between the autonomous aerial device and the ground device in the estimation of the relative position.

[0015] In some embodiments, the processor is configured to transform the relative position into a map frame to obtain a pose of the autonomous aerial device in a map.

[0016] In some embodiments, the one or more measurements obtained from the IMU of the autonomous aerial device comprises a relative roll angle and / or a relative pitch angle of the autonomous aerial device.

[0017] In some embodiments, the one or more measurements obtained from the IMU of the ground device comprises a relative roll angle and / or a relative pitch angle of the ground device.

[0018] In some embodiments, the at least one processor is configured to obtain a current positional dataset of the ground device, the current positional dataset comprises a fourdimensional (4D) pose data of the ground device.

[0019] In some embodiments, the positional dataset of the autonomous aerial device comprises a six-dimensional (6D) pose data of the relative position of the autonomous aerial device with respect to the ground device.

[0020] In some embodiments, the 6D pose data is generated by a deep neural network estimation algorithm.

[0021] In some embodiments, the one or more visual markers comprises at least one fiducial marker.

[0022] In some embodiments, the at least one fiducial marker includes a AprilTag marker.

[0023] In some embodiments, the autonomous aerial device comprises a tethered drone.

[0024] According to another aspect of the present disclosure, there is provided a tethered drone and ground robot tandem navigation system, the system comprising a sensor configured to detect one or more visual markers positioned on a ground robot; at least one processor, the at least one processor configured to: obtain, from an inertial measurement unit (IMU) of the tethered drone, one or more measurements of the tethered drone; obtain, from an inertial measurement unit (IMU) of the ground robot, one or more measurements of the ground robot; and generate a plurality of rotation matrices based on the obtained one or more measurements from the tethered drone and one or more measurements from the ground robot; detect, using a marker-based detection method, image data associated with one or more visual markers of the ground device, and estimate a positional dataset of the tethered drone based on the image data; and estimate the relative position based on the plurality of rotation matrices, the positional dataset of the tethered drone. In some embodiments, the at least one processor is further configured to estimate a heading, in addition to the relative position, based on the plurality of rotation matrices and the positional dataset of the tethered drone. The heading and the relative position may form a relative pose of the tethered drone with respect to the ground robot.

[0025] In some embodiments, the plurality of rotation matrices comprises a first rotation matrix and second rotation matrix, wherein the first rotation matrix is associated with the one or more measurements of the IMU of the autonomous aerial device and the second rotation matrix is associated with the one or more measurements of the IMU of the ground device.

[0026] In some embodiments, the sensor comprises an image capturing device, the image capturing device orientated to the ground device in a manner to detect the one or more visual markers.

[0027] According to another aspect of the present disclosure there is provided a method for estimating a relative position of an autonomous aerial device with respect to a ground device, the method comprising: obtaining, from an inertial measurement unit (IMU) of the autonomous aerial device, one or more measurements of the autonomous aerial device; obtaining, from an IMU of the ground device, one or more measurements of the ground device, and generating a plurality of rotation matrices based on the one or more measurements of the autonomous aerial device and the one or more measurements of the ground device; detecting, using a markerbased detection method, image data associated with one or more visual markers of the ground device, and estimating a positional dataset of the autonomous aerial device based on the imagedata; and estimating the relative position based on the plurality of rotation matrices and the positional dataset of the autonomous aerial device. In some embodiments, the method further comprise a step of estimating a heading, in addition to the relative position, based on the plurality of rotation matrices and the positional dataset of the autonomous aerial device. The heading and the relative position may form a relative pose of the autonomous aerial device with respect to a ground device.

[0028] In some embodiments, generating the plurality of rotation matrices comprises generating a first rotation matrix and generating second rotation matrix, wherein the first rotation matrix is associated with the one or more measurements of the IMU of the autonomous aerial device and the second rotation matrix is associated with the one or more measurements of the IMU of the ground device.

[0029] In some embodiments, estimating the relative position further comprises generating a shared coordinate system between the autonomous aerial device and the ground device.

[0030] In some embodiments, the method further comprises transforming the relative position into a map frame to obtain a pose of the autonomous aerial device in the map frame.

[0031] In some embodiments, the one or more measurements obtained from the IMU comprises a relative roll angle and / or a relative pitch angle of the autonomous aerial device.

[0032] In some embodiments, the method further comprises obtaining a current positional dataset of the ground device, the current positional dataset comprises a four-dimensional (4D) pose data of the ground device.

[0033] In some embodiments, the positional dataset of the autonomous aerial device comprises a six-dimensional (6D) pose data of the relative position of the autonomous aerial device with respect to the ground device.

[0034] In some embodiments, the 6D pose data is generated by a deep neural network estimation algorithm.

[0035] In some embodiments, the one or more visual markers comprises at least one fiducial marker.

[0036] In some embodiments, the at least one fiducial marker includes a AprilTag marker.

[0037] In some embodiments, the autonomous aerial device comprises a tethered drone.

[0038] According to another aspect of the present disclosure there is provided a computer-readable medium comprising program instructions, which, when executed by one or more processors, cause the one or more processors to perform any one of the methods as described.BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The disclosure will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and the accompanying drawings, in which:- FIG. 1 shows an arrangement comprising a tethered drone and ground robot performing tandem operation in simulation (left) and in real world (right).- FIG. 2 shows a system architecture diagram of a tethered drone and ground robot tandem navigation system according to some embodiments.- FIG. 3 illustrates simulation results comparing purely AprilTag-based estimation (left panel) with the proposed approach (right panel).- FIG. 4 illustrates a general flowchart of a method for estimating a relative position of an autonomous aerial device with respect to a ground device.DETAILED DESCRIPTION

[0040] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. Other embodiments may be utilized and structural, logical changes may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0041] Embodiments described in the context of one of the systems or methods are analogously valid for the other systems or methods.

[0042] Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments,even if not explicitly described in these other embodiments. Furthermore, additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0043] In the context of some embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0044] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0045] As used herein, the term “associate”, “associated”, and “associating” indicate a defined relationship (or cross-reference) between two items.

[0046] As used herein, the term “processor(s)” includes one or more electrical circuits capable of processing data, i.e., processing circuits. A processor may include analog circuits or components, digital circuits or components, or hybrid circuits or components. Any other kind of implementation of the respective functions which will be described in more detail below may also be understood as a "circuit" in accordance with an alternative embodiment. A digital circuit may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, or a firmware.

[0047] As used herein, “memory” may be understood as a non-transitory computer-readable medium in which data or information can be stored for retrieval. References to “memory” included herein may thus be understood as referring to volatile or non-volatile memory, including random access memory (“RAM”), read-only memory (“ROM”), flash memory, solid-state storage, magnetic tape, hard disk drive, optical drive, etc., or any combination thereof. Furthermore, it is appreciated that registers, shift registers, processor registers, data buffers, etc., are also embraced herein by the term memory. It is appreciated that a single component referred to as “memory” or “a memory” may be composed of more than one different type of memory, and thus may refer to a collective component including one or more types of memory. It is readily understood that any single memory' component may be separated into multiple collectively equivalent memory components, and vice versa. Furthermore, while memory may be depicted as separate from one or more other components (such as in the drawings), it is understood that memory may be integrated within another component, such as on a common integrated chip.

[0048] As used herein, the term “configured to” broadly refers to how an apparatus, a system, or a component may be arranged, designed, or programmed to perform a specificfunction. For example, if a processor is "configured to transmit data," it means the processor comprises the hardware (e.g., circuitry) and / or software (e.g., executable software code) for performing the data transmission function.

[0049] As used herein, the term “autonomous aerial vehicle” refers to any unmanned aerial vehicle (UAV) or aerial system capable of sustained flight without direct human control onboard. This includes, but is not limited to, aircraft that are remotely piloted, autonomously operated, or semi-autonomous, and may utilize various propulsion systems, navigation technologies, and payload configurations. Autonomous aerial vehicles may include one or more drones, which may be designed for civilian, commercial, industrial, scientific, agricultural, recreational, and / or military applications. The term “drone” encompasses aerial platforms of all sizes — from micro and nano drones to large-scale unmanned aircraft — across fixed-wing, rotary-wing, hybrid, and novel configurations.

[0050] As used herein, the term “data” may be understood to include information in any suitable analog or digital form, for example, provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. The term data, however, is not limited to the aforementioned examples and may take some forms and represent any information as understood in the art.

[0051] As used herein, the term “obtain” refers to the processor which actively obtains the inputs, or passively receives inputs from one or more sensors or data source. The term obtain may also refer to a processor, which receives or obtains inputs from a communication interface, e.g., a user interface. The processor may also receive or obtain the inputs via a memory, a register, and / or an analog-to-digital port.

[0052] As used herein, the term “module” refers to, or forms part of, or include an Application Specific Integrated Circuit (ASIC); an electrical / electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module may include memory (shared, dedicated, or group) that stores code executed by the processor.

[0053] As used herein, the term “marker-based detection method” broadly refers to any computer vision technique or system that identifies, localizes, and optionally estimates the pose of predefined visual patterns — referred to as markers — within an image or video stream. Thesemarkers may consist of binary, alphanumeric, geometric, or symbolic designs optimized for reliable detection under varying environmental conditions. In some embodiments, the markers may include passive markers (e.g., flat color patterns) or active markers (e.g., powered light patterns). The detection process may involve image preprocessing, feature extraction, pattern recognition, and pose estimation algorithms. Marker-based detection method and / or systems may be used in applications such as robotics, augmented reality, autonomous navigation, object tracking, and industrial automation. Non-limiting examples include AprilTag, ArUco, QR codes, Data Matrix codes, CCTag, and other fiducial marker systems.

[0054] In the following, embodiments will be described in detail.

[0055] FIG. 1 shows an autonomous aerial device and ground device arrangement in the form of a drone 102, a tether cable 104 and a ground robot 106. The figure is divided into two panels: the left panel shows a computer-generated model / simulation of the system, and the right panel shows a real- world implementation.

[0056] The tethered drone 102 may be a quadrotor drone, operating in coordination with the ground-based robot 106. The drone 102 may be equipped with an image capturing device (not shown), the image capturing device orientated to capture one or more visual markers 108, the one or more visual markers 108 may be positioned on the ground robot 106.

[0057] In some embodiments, the drone 102 is a multi-rotor unmanned aerial vehicle (UAV) equipped with four propellers. The drone 102 may be capable of vertical take-off, hovering, and precise manoeuvring.

[0058] In some embodiments, the tether cable 104 may be a flexible cable connecting the drone 102 to the ground robot 106. The tether cable 104 may be arranged to provide power, such as continuous electrical power supply and / or high-bandwidth data transmission. In some embodiments, the tether cable 104 may include fiber optics, copper conductors, or hybrid materials.

[0059] In some embodiments, the ground robot 106 may be in the form of a ground-based vehicle having a mobile base 122, an electronics housing unit 124, and an antenna or communication module 126.

[0060] The mobile base 122 may be a four-wheeled robotic platform capable of autonomous or remote-controlled navigation. The mobile base 122 may serve as the mobile anchor point for the tethered drone. In some embodiments, the mobile base 122 may house power systems, processing units, and communication modules.

[0061] The electronics housing unit 124 may be an enclosure mounted on the mobile base 122. The electronics housing unit 124 may contain batteries, processors (including IMU), and tether management systems. Tn some embodiments, the electronics housing unit 124 may include cooling systems and shielding for electromagnetic interference.

[0062] The antenna 126 may include a cylindrical protrusion mounted atop the electronics housing unit 124. The antenna 126 may facilitate wireless communication with external systems (e.g., base station, other robots), and may support different communication protocols such as global positioning system (GPS), Long Term Evolution (LTE), Wireless Fidelity (WiFi), or custom radio frequency (RF) protocols.

[0063] The visual marker 108 may be a fiducial marker positioned (e.g. affixed) to a surface of the ground robot 106, typically on the top or a side of the electronics housing unit 124 to facilitate image capture by the image capturing device.

[0064] In some embodiments, the visual marker 108 may include an encoding of a unique identifier (ID) and geometric pattern detectable by the drone’s image capturing device or onboard camera.

[0065] The visual marker 108 may be used for relative pose estimation between the drone and the ground vehicle, and may be used to facilitate high-precision localization in environments where GPS is unavailable or unreliable. In some embodiments, the visual marker 108 may be a AprilTag, which serves as a visual anchor for the drone’s onboard vision system. By detecting and decoding the tag, the drone 102 can compute its relative 4D pose (x, y, z, y) with respect to the ground robot 106. This relative pose may then be transformed into a global pose using the ground robot 106 SL AM-based localization.

[0066] In general, the tethered drone system enables extended-duration aerial operations without reliance on onboard batteries. The ground robot 106 autonomously follows or positions itself to optimize the flight path of the drone 102. The tether cable 104 facilitates uninterrupted power and data exchange, allowing the drone 102 to perform tasks such as surveillance, mapping, or communication relay.

[0067] According to an aspect of the present disclosure, there is provided a server apparatus for estimating a relative position of an autonomous aerial device with respect to a ground device, the server apparatus comprising at least one processor configured to: obtain, from the ground device, a current positional dataset of the ground device; obtain, from an inertial measurement unit (IMU) of the autonomous aerial device, one or more measurements of the autonomousaerial device; obtain, from an IMU of the ground device, one or more measurements of the ground device; generate a plurality of rotation matrices based on the one or more measurements of the autonomous aerial device and the one or more measurements of the ground device; detect, using a marker-based detection method, image data associated with one or more visual markers of the ground device, and estimate a positional dataset of the autonomous aerial device based on the image data; and estimate the relative position based on the plurality of rotation matrices and the positional dataset of the autonomous aerial device. The server apparatus may be arranged in data communication with a flight controller, or form part of the flight controller, and arranged in data communication with an inertial measurement unit (IMU) of the aerial autonomous device and / or the IMU of the ground device. In some embodiments, the plurality of rotation matrices comprises a first rotation matrix and second rotation matrix, wherein the first rotation matrix is generated based on the one or more measurements of the IMU of the autonomous aerial device, and the second rotation matrix is generated based on the one or more measurements of the IMU of the ground device. In some embodiments, the at least one processor is further configured to estimate a heading, in addition to the relative position, based on the plurality of rotation matrices and the positional dataset of the tethered drone. The heading and the relative position may form a relative pose of the tethered drone with respect to the ground robot.

[0068] Referring to FIG. 2, there is provided a system architecture of a tethered drone coordination system 200, the coordination system 200 comprising a ground control station 202, a ground robot 204, and a tethered drone 206, arranged in signal and / or data communication with one another. Such signal and / or data communication may facilitate interconnectivity via control and data pathways.

[0069] The ground control station 202 may be a computing device (e.g., laptop or tablet) used to issue high-level mission commands to the ground robot 204, and the tethered drone 206. In some embodiments, the high-level mission commands issued to the tethered drone 206 may include commands such as, but not limited to, a take-off command, a loiter command, and a land command. The computing device may be configured to send the commands to both the ground robot 204, and the tethered drone 206.

[0070] In some embodiments, the ground control station 202 may interface with the other components 204, 206 via wireless communication protocols.

[0071] The ground root 204 may function as a mobile robotic platform equipped with localization and control modules.

[0072] Tn some embodiments, the ground robot 204 may include a Simultaneous Localization and Mapping (SLAM) module 211. The SLAM module 211 may be configured to perform simultaneous localization and mapping to determine a current positional dataset of the ground robot, for example, the ground robot’s current 4D pose (x, y, z, yaw) in an environment. The ground robot 204 may include an IMU.

[0073] The ground robot 204 may also include a waypoint controller 212. The waypoint controller 212 may be configured to receive the relatively high-level commands from the ground control station 202 and translate the relatively high-level commands into actuator-level control signals based on the current pose of the ground robot 204.

[0074] The tethered drone 206 may be an aerial quadrotor system physically connected to the ground robot 204 via a tether cable 230.

[0075] The tethered drone 206 includes a flight controller, such as a Pixhawk flight controller 221. The Pixhawk flight controller 221 may include one or more microprocessors, inertial measurement units (IMUs), and sensor interfaces configured to receive data from onboard sensors, including accelerometers, gyroscopes, magnetometers, barometers, and / or GPS modules. Based on the sensor data, the Pixhawk flight controller 221 may be configured to execute flight control algorithms to determine attitude, position, and velocity, and outputs corresponding control signals to actuators such as motors or servos. The Pixhawk platform may support modular integration with various hardware and software components through standardized communication protocols such as MAVLink, enabling flexible adaptation for different vehicle configurations and mission requirements. The Pixhawk flight controller 221 may be configured to receive / obtain a set of desired 4D velocity data and a set of current 4D pose data, and output the IMU data (roll, pitch) for control of the drone, including control signals to stabilize and navigate the drone.

[0076] The tethered drone 206 includes a drone waypoint controller 222. The drone waypoint controller 222 is configured to compute the set of desired 4D velocity data (x, y, z, yaw) based on pose data and TMU feedback, the desired 4D velocity data forming an input to the Pixhawk flight controller 221.

[0077] The tethered drone 206 further comprises a transformation handler module 223 to convert relative 6D pose (xi, yi, zi, X2, y2, Z2) of the drone into a current 4D pose for the dronewaypoint Controller 222. The transformation handler module 223 may be configured to obtain or receive, from an IMU of the autonomous aerial device (e.g. output of the Pixhawk flight controller 221 ), one or more measurements of the autonomous aerial device; obtain or receive, from an IMU of the ground device, one or more measurements of the ground device; and generate a plurality of rotation matrices based on the one or more measurements of the autonomous aerial device and the one or more measurements of the ground device. In some embodiments, the SLAM module 211 may be configured to output a current positional dataset of the ground device, for example, a current 4D pose data of the ground device.

[0078] The tethered drone 206 includes a visual marker detection module, such as an AprilTag detection module 224. The AprilTag detection module 224 may be used to process one or more RGB images received from the drone’s onboard camera to detect AprilTags affixed to the ground robot 204 to enable the computation of the relative pose between the drone 206 and the ground robot 204. In some embodiments, each of the RGB images may have a resolution of 1920 x 1200 pixels.

[0079] The tether cable 230 provides a physical connection between the ground robot 204 and the tethered drone 206, to provide power and data communication to the drone, enabling extended flight duration and real-time control.

[0080] In some embodiments, the flight controller may be configured to implement an autonomous control system for aerial-ground robotic coordination via relative 4D pose estimation of the tethered drone 206. Assuming the tethered drone 206 is quadrotor drone, the control framework that enables autonomous navigation and coordination with a ground robot may comprise a control input, the control input defined as a vector control of regular drones achieved through the generation of the collective thrust or force (u, ) and the moments around each axis (u?, U3, U4) by the actuators, which form the control input mathematically expressed in Equation (1) as follows:u e IB!4(1) wherein ui is the collective thrust, U2, U3, U4 are the moments around the roll, pitch, and yaw axes respectively.

[0081] The drone state, comprising the 12 states of a quadrotor, may be mathematically expressed in Equation (2) as follows:x e IR12(2)

[0082] In some embodiments, the control input u can be written as algebraic functions of differentially flat outputs selected in, and mathematically expressed in Equation (3) and Equation (4) as follows:

[0083] where x, y, z stand for the position of the drone 206 in three axes, and \| / is the heading of the drone. To achieve the autonomous control of the drone 206 in-tandem with the ground robot 204, precise feedback may be obtained on the four states by estimating the relative 4D pose (.r, y, z, i / f) of the drone 206 with respect to the ground robot 204. Since the ground robot 204 pose in a map is known through simultaneous localization and mapping (SLAM), the drone pose in the map may then be calculated.

[0084] As may be appreciated in FIG. 2, a marker-based detection method (AprilTag) combined with IMU data integration for 4D relative pose estimation of the tethered drone may be achieved. This enables the drone 206 to operate autonomously including auto-takeoff, loiter, and precise landing on the ground robot 204.

[0085] The AprilTag detection method may be a widely adopted fiducial marker system that is used in many robotic applications such as grasping, object tracking, precise landing, etc. The main working principle of AprilTag may comprise three steps as follows.

[0086] Step 1: detecting distinct visual features provided by the white and black regions on a tag;

[0087] Step 2: measuring the tag size and position by detecting its boundaries in the image frame;

[0088] Step 3: translating them in the local map frame by using the camera intrinsic and a priori known physical tag size.

[0089] This process enables relative pose estimation of a camera with respect to a tag, which can be then transformed into the drone's relative pose with respect to the ground robot. The precision of such estimation relies on ambient lighting, image resolution, proper camera calibration, and proper tag family selection. In some embodiments, the 36hl8 tag family may be selected as it strikes a favorable balance between pose accuracy and max distance that it can be detected from. In some embodiments, the landing pad on the ground robot may be equipped with five tags comprising a relatively small tag with an edge length of 2 centimeters (cm),which is used when the drone is resting on the pad, four relatively larger tags with an edge length of 16 cm, which is used for auto-takeoff, loiter, and precise landing.

[0090] Although the AprilTag Detection Module 224 may provide relative pose estimation between the drone and the ground robot using visual markers, it was discovered that this method may suffer from instability at higher altitudes, primarily due to inaccurate estimation of roll (<|>) and pitch (0) angles. Such angular inaccuracies are amplified as the drone moves away from the tag, resulting in significant errors in the estimated x and y positions.

[0091] Tn some embodiments, one or more measurements from the IMU of the Pixhawk flight controller 221 includes orientation data, the orientation data may include data such as roll and pitch data of the drone, or heading related data, may be used for integration with the output received from the AprilTag, which may include an estimated relative 6D pose data. The roll and pitch data, together with the estimated relative 6D pose data, and the 4D pose data of the ground robot obtained from the SLAM module, may form the input for transformation, as a way to mitigate the drawback. In particular, the system integrates IMU data from both the drone 206, and the ground robot 204, to correct the raw 6D pose estimation of AprilTag.

[0092] In some embodiments, the raw 6D pose estimation, denoted praw, estimated by the AprilTag, may be translated into a shared coordinate system between the autonomous aerial device and the ground device, such as a vehicle-carried frame pvc-Dof the drone 206, to eliminate the inaccuracy caused by the raw roll and pitch data <>rawand 0raw. The translation may be mathematically expressed in Equation (5), as follows.Pvc-D—RQ Praw (5) wherein Rois a first rotation matrix formed using the roll <>IMU-Dand pitch 0IMU-Ddata from the IMU of the drone 206, denoted as from the drone's IMU and the yaw data i>rawfrom the AprilTag detection or estimation.

[0093] The pose pvc-Dmay then be transformed into the vehicle-carried frame for the ground robot pvc-GR,the transformation mathematically expressed in Equation (6), as follows.where Rxis a second rotation matrix formed using the orientation data, such as roll data < > IMU-GR from the ground robot’s IMU and the pitch data 0JMU-GR from the ground robot's IMU. This transformation eliminates or mitigates sudden jumps in pose estimation caused by abrupt changes in the AprilTag’ s orientation due to terrain irregularities. For example, whereby AprilTags change their <p and 0 angles abruptly.

[0094] After the pose has been transformed, and since the current positional dataset of the ground robot, i.e. 4D pose of the ground robot is known in a map via the SLAM module 211, the drone’s global pose pmap-Dma. V be calculated and mathematically expressed in Equation (7), as follows.Pmap-D—T Pvc-GR C )

[0095] where T is a transformation matrix derived from the SLAM-based global pose of the ground robot comprising the parameters x, y, z, \| / by SLAM. All these operations are handled by transformation handler module 223, which integrates IMU data from both platforms, applies rotation matrices to correct visual pose estimates, and outputs accurate 4D pose (x, y, z, ) for the drone waypoint controller 222.

[0096] Tn studies carried out, it was found that purely April Tag-based estimation provides stable localization only for low altitudes. This is due to inaccurate estimation of relative roll (4>) and pitch (0) angles with respect to the AprilTag. Solely relying on visual features causes inaccuracy to such estimation. IMU data was instead leveraged on for the estimation of relative roll and pitch angles. Precise estimation of these angles is found to improve the 4D pose (x, y, z, \| / ) estimation of the drone substantially because for marker-based detection methods, they are tightly coupled as the reference frame is the tag itself. Such a coupling effect becomes more visible as the drone moves away from the AprilTag and flies at higher altitudes. A small deviation in <[> or 0 (a few degrees) can result in meters of error in x and y estimations in such conditions.

[0097] In some embodiments, the system 200 may comprise several robot operating system (ROS) packages which may be written in different programming language, such as Python and C++. These packages communicate over the same ROS network whose master is running on the ground robot's onboard computer, which may be an Advantech MIO5393 board with an Intel Xeon E-2276ME processor and 64GB RAM. The same computer may also be responsible for SLAM of the ground robot to inform the robot team on their localization with respect to the environment. The drone's onboard computer, which may include an Intel i7- 1165G7 processor and 32GB RAM, runs the relative localization as well as the high-level waypoint controller which sends commands to a Pixhawk flight controller flashed with a PX4 firmware. In some embodiments, both the computers may be installed with Linux operating systems, and they communicate over a wireless Wi-Fi network.

[0098] The architecture remains the same for simulation studies based on software-in-the-loop (SITL) Gazebo simulations by PX4. Dell Alienware xl7 laptop with an Intel i7-11800H processor and 32GB RAM to host the simulations. It may be appreciable that like the real robot setup, the computer used for simulation has a Linux operating system.

[0099] Simulation results and discussion - Flight tests both in simulations and in real world were carried out. The benchmarked approach with respect to purely AprilTag-based approach at altitude levels ranging from 3 meters (m) to 14 m in simulations are shown in FIG. 3. Successful auto-takeoff, hover, and precise landing with the drone 206 were demonstrated. There are also figures showing successful ground robot following in real world, both indoors and outdoors. It may be appreciable that FIG. 3 shows the tethered drone struggling to hover at 5 m (below the target level marked ‘T’) with purely AprilTag-based estimation (left panel marked ‘A’) and hovering in a very stable manner at 10 m (within the target level marked ‘T’) with the proposed approach (right panel marked ‘B’).

[0100] In some embodiments, benchmarking tests arc conducted in simulation environments to evaluate the performance of the proposed tethered drone localization system against conventional AprilTag-only methods. The results of these tests are summarized in Table 1, which presents the mean-squared 3D trajectory error at various drone altitudes during hovering operations.

[0101] The benchmarking procedure involves commanding the drone to hover at fixed altitudes ranging from 1 meter to 14 meters above the ground robot, while tracking its 3D trajectory. The drone’s estimated pose is compared against the ground truth to compute the mean-squared error in position.

[0102] As shown in Table 1, both the AprilTag-only method and the proposed IMU-enhanced method maintain low trajectory errors (within a few centimeters) at altitudes up to 5 meters. However, beyond this threshold, the performance of the AprilTag-only method degrades significantly. At 6 meters, the drone begins to exhibit erratic behavior due to unreliable pose estimation, with localization errors exceeding acceptable thresholds. This instability is attributed to the increasing inaccuracy in estimating the drone’s roll (<])) and pitch (0) angles solely from visual features, which are tightly coupled with positional estimates in marker-based detection methods or systems.

[0103] In contrast, the proposed method, which integrates IMU data from both the drone and the ground robot, maintains stable and accurate localization up to 12 meters of altitude,with trajectory errors still within a few centimeters. Even at the maximum tested altitude of 14 meters, the average error remains at approximately 16 centimeters, which is significantly lower than the error observed with AprilTag-only estimation at the altitude of 6 meters.

[0104] These results demonstrate the superior performance and robustness of the proposed system, particularly in high-altitude scenarios where visual-only methods may fail. The integration of IMU data into the transformation pipeline, as described in the Transformation Handler module 223 of FIG. 2, enables precise correction of angular deviations and improves overall pose estimation fidelity.

[0105] The benchmarking results validate the system’ s capability to support extended-range aerial operations with high localization accuracy, making it suitable for applications in surveillance, mapping, and autonomous coordination with ground robots in GPS-denied environments.

[0106] Table 1: Mean-squared 3D trajectory error results in meters (averaged over five inns) while hovering with purely AprilTag-bascd and with the proposed integrated approach.Table 1

[0107] In the present disclosure, the problem of relative localization between robots, specifically emphasizing tandem operations involving a tethered aerial drone and a ground robot, as illustrated in FIG. 1, was addressed. Unlike conventional approaches that rely on external infrastructure or heavy sensor payloads, a lightweight, infrastructure-independent method that achieves accurate relative localization using vision-based techniques is proposed. To this end, the drone 206 may be equipped with a compact, downward-facing camera that captures aerial images of the ground robot 204 and estimates its relative pose. The proposed approach combines 6-dimensional pose estimation using visual markers, such as AprilTags,with inertial data from onboard IMUs to enhance accuracy and robustness. This fusion of visual and inertial information enables reliable localization even in challenging environments.

[0108] The proposed system is designed to operate without dependence on GNSS or external beacons, making it suitable for deployment in unstructured or GPS-denied environments, including security-critical missions. Benchmarking against a widely used AprilTag-only method demonstrates that the proposed approach significantly outperforms existing techniques, particularly in vertical flight scenarios. While the AprilTag-only method fails to maintain stable localization beyond 5 meters (m) altitude, the proposed system supports flights exceeding 10 meters with minimal trajectory error. Real-world experiments conducted both indoors and outdoors further validate the system’s performance.

[0109] It may be appreciable that although AprilTags are used for benchmarking, the proposed framew'ork is not limited to this marker type. It is compatible with alternative visual fiducials such as ArUco markers, infrared tags (or examples mentioned in

[0053] ), or deep learning-based 6D pose estimation methods. This flexibility allows the system to be adapted to various operational contexts and sensor configurations. Furthermore, the localization framework can be integrated with intelligent path planning algorithms to enable autonomous navigation in cluttered environments and under varying lighting conditions. This includes strategies for collision avoidance and tether management to prevent entanglement during flight.

[0110] In some embodiments, the abovementioned localization approach for a tethered drone & ground robot platform comprises a method for positioning an autonomous aerial device. The method comprises estimating a pose of a ground device in a map; detecting one or more visual features of the ground device by at least one sensor of the autonomous aerial device; estimating a first relative pose of the autonomous aerial device based on the one or more visual features; obtaining one or more measurements of the autonomous aerial device from an inertial measurement unit (IMU); calculating a second relative pose of the autonomous aerial device based on the first relative pose of the autonomous aerial device, the one or more measurements obtained from the IMU and the pose of the ground device; and transforming the second relative pose into a map frame to obtain a pose of the autonomous aerial device in the map.

[0111] In one embodiment of the abovementioned method, the one or more measurements obtained from the IMU of the autonomous aerial device comprises a relative roll angle and / or a relative pitch angle of the autonomous aerial device.

[0112] In one embodiment of the abovementioned method, the one or more visual features comprises one or more markers attached to the ground device, and wherein the at least one sensor of the autonomous aerial device comprises a downward-facing camera.

[0113] In one embodiment of the abovementioned method, the autonomous aerial device comprises a tethered drone.

[0114] According to another aspect of the present disclosure, there is provided a method 400 for estimating a relative position of an autonomous aerial device with respect to a ground device, the method comprising:

[0115] Step S401: obtaining, from an inertial measurement unit (IMU) of the autonomous aerial device, one or more measurements of the autonomous aerial device;

[0116] Step S402: obtaining, from an IMU of the ground device, one or more measurements of the ground device;

[0117] Step S403: generating a plurality of rotation matrices based on the one or more measurements of the autonomous aerial device and the one or more measurements of the ground device;

[0118] Step S404: detecting, using a marker-based detection method, image data associated with one or more visual markers of the ground device, and estimating a positional dataset of the autonomous aerial device based on the image data; and

[0119] Step S405: estimating the relative position based on the plurality of rotation matrices and the positional dataset of the autonomous aerial device.

[0120] In some embodiments, the method comprises obtaining a current positional dataset of the ground device, wherein the current positional dataset of the ground device comprises a four-dimensional (4D) pose data of the ground device. In some embodiments, the method comprises estimating a heading, in addition to the relative position, based on the plurality of rotation matrices and the positional dataset of the tethered drone. The heading and the relative position may form a relative pose of the tethered drone with respect to the ground robot.

[0121] In some embodiments, the estimating the relative position further comprises generating a shared coordinate system between the autonomous aerial device and the ground device.

[0122] In some embodiments, the method further comprises transforming the relative position into a map frame to obtain a pose of the autonomous aerial device in the map.

[0123] In some embodiments, the one or more measurements obtained from the IMU comprises a relative roll angle and / or a relative pitch angle of the autonomous aerial device.

[0124] In some embodiments, the current positional dataset of the ground device comprises a four-dimensional (4D) pose data of the ground device.

[0125] In some embodiments, the positional dataset of the autonomous aerial device comprises a six-dimensional (6D) pose data of the relative position of the autonomous aerial device with respect to the ground device.

[0126] In some embodiments, the 6D pose data is generated by a deep neural network estimation algorithm.

[0127] In some embodiments, the one or more visual markers comprises at least one fiducial marker.

[0128] In some embodiments, the at least one fiducial marker includes a AprilTag marker.

[0129] In some embodiments, the autonomous aerial device comprises a tethered drone.

[0130] In some embodiments, the plurality of rotation matrices comprises a first rotation matrix and second rotation matrix, wherein the first rotation matrix is generated based on the one or more measurements of the IMU of the autonomous aerial device, and the second rotation matrix is generated based on the one or more measurements of the IMU of the ground device.

[0131] According to another aspect of the present disclosure there is provided a computer-readable medium comprising program instructions, which, when executed by one or more processors, cause the one or more processors to perform the method as described.

[0132] It may be appreciable that conventional approaches to integrating inertial measurement unit (IMU) data with AprilTag-based pose estimation typically involve utilizing IMU signals to stabilize roll and pitch in the horizontal plane, while relying on AprilTag -derived position and yaw estimates to guide the drone's spatial navigation. However, empirical testing revealed a critical limitation: drones operating under this conventional scheme exhibited instability beyond an altitude of approximately 4-5 meters, particularly in environments lacking GNSS or LiDAR support. Given that the present system of the disclosure is designed to maintain stable flight at altitudes of at least 10 meters above a ground-based robot, a fundamentally different integration strategy was required.

[0133] Through extensive experimentation — conducted daily over several months, typically spanning 8-10 hours per day — the present disclosure was conceived that synergistically leverages both IMU and AprilTag data in a non-obvious manner. Theintegration, as detailed in the IMU integration section, departs from conventional paradigms and enables robust, high-altitude flight stability under constrained sensing conditions.

[0134] While the use of multiple AprilTags — specifically, bundles of four — may initially appear to address occlusion caused by tethering cables, their inclusion is primarily motivated by the enhanced reliability of pose estimation. Empirical evidence and prior studies confirm that AprilTag bundles yield more robust and accurate pose data than single-tag configurations, regardless of tethering.

[0135] It may be further appreciable that the integration of IMU data with AprilTag-based estimation remains effective even when implemented with a single AprilTag and in nontethered drone scenarios. Such an arrangement demonstrates the versatility of the present disclosure across various deployment contexts.

[0136] While the disclosure has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims. The scope of the disclosure is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

CLAIMS1. A server apparatus for estimating a relative position of an autonomous aerial device with respect to a ground device, the server apparatus comprising at least one processor configured to:obtain, from an inertial measurement unit (IMU) of the autonomous aerial device, one or more measurements of the autonomous aerial device;obtain, from an inertial measurement unit (IMU) of the ground device, one or more measurements of the ground device;generate a plurality of rotation matrices based on the one or more measurements of the autonomous aerial device and the one or more measurements of the ground device;detect, using a marker-based detection method, image data associated with one or more visual markers of the ground device, and estimate a positional dataset of the autonomous aerial device based on the image data; andestimate the relative position based on the plurality of rotation matrices and the positional dataset of the autonomous aerial device.

2. The server apparatus of claim 1, wherein the at least one processor is configured to generate a shared coordinate system between the autonomous aerial device and the ground device in the estimation of the relative position.

3. The server apparatus of claim 2, wherein the processor is configured to transform the relative position into a map frame to obtain a pose of the autonomous aerial device in the map.

4. The server apparatus of any one of claims 1 to 3, wherein the one or more measurements obtained from the IMU comprises a relative roll angle and / or a relative pitch angle of the autonomous aerial device.

5. The server apparatus of any one of claims 1 to 4, at least one processor is configured to obtain a current positional dataset of the ground device, wherein the current positional dataset of the ground device comprises a four-dimensional (4D) pose data of the ground device.

6. The server apparatus of any one of claims 1 to 5, wherein the positional dataset of the autonomous aerial device comprises a six-dimensional (6D) pose data of the relative position of the autonomous aerial device with respect to the ground device.

7. The server apparatus of claim 6. wherein the 6D pose data is generated by a deep neural network estimation algorithm.

8. The server apparatus of any one of claims 1 to 7, wherein the one or more visual markers comprises at least one fiducial marker.

9. The server apparatus of claim 8, wherein the at least one fiducial marker includes a AprilTag marker.

10. The server apparatus of any one of claims 1 to 9, wherein the autonomous aerial device comprises a tethered drone.

11. A tethered drone and ground robot tandem navigation system, the system comprising a sensor configured to detect one or more visual markers positioned on a ground robot; at least one processor, the at least one processor configured to:obtain, from an inertial measurement unit (IMU) of the tethered drone, one or more measurements of the tethered drone;obtain, from an IMU of the ground robot, one or more measurements of the ground robot;generate a plurality of rotation matrices based on the one or more measurements of the tethered drone and the one or more measurements of the ground robot;detect, using a marker-based detection method, image data associated with one or more visual markers of the ground robot, and estimate a positional dataset of the tethered drone based on the image data; andestimate a relative position of the tethered drone with respect to the ground robot based on the plurality of rotation matrices and the positional dataset of the tethered drone.

12. The system of claim 11, wherein the sensor comprises a downward-facing camera.

13. A method for estimating a relative position of an autonomous aerial device with respect to a ground device, the method comprising:obtaining, from an inertial measurement unit (IMU) of the autonomous aerial device, one or more measurements of the autonomous aerial device;obtaining, from an IMU of the ground device, one or more measurements of the ground device;generating, a plurality of rotation matrices based on the one or more measurements of the autonomous aerial device and the one or more measurements of the ground device;detecting, using a marker-based detection method, image data associated with one or more visual markers of the ground device, and estimating a positional dataset of the autonomous aerial device based on the image data; andestimating the relative position based on the plurality of rotation matrices and the positional dataset of the autonomous aerial device.

14. The method of claim 13, wherein the estimating the relative position further comprises generating a shared coordinate system between the autonomous aerial device and the ground device.

15. The method of claim 14, further comprises transforming the relative position into a map frame to obtain a pose of the autonomous aerial device in the map.

16. The method of any one of claims 13 to 15, wherein the one or more measurements obtained from the IMU comprises a relative roll angle and / or a relative pitch angle of the autonomous aerial device.

17. The method of any one of claims 13 to 16, further comprises obtaining a current positional dataset of the ground device, wherein the current positional dataset of the ground device comprises a four-dimensional (4D) pose data of the ground device.

18. The method of any one of claims 13 to 17, wherein the positional dataset of the autonomous aerial device comprises a six-dimensional (6D) pose data of the relative position of the autonomous aerial device with respect to the ground device.

19. The method of claim 18, wherein the 6D pose data is generated by a deep neural network estimation algorithm.

20. The method of any one of claims 13 to 19, wherein the one or more visual markers comprises at least one fiducial marker.

21. The method of claim 20, wherein the at least one fiducial marker includes a AprilTag marker.

22. The method of any one of claims 13 to 21, wherein the autonomous aerial device comprises a tethered drone.

23. A computer- readable medium comprising program instructions, which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 13 to 22.

Images