System and method for BIO-inspired visually-guided multi-agent coordination
A bio-inspired visual processing system allows mobile agents to coordinate using passive sensing and sparse networks, addressing scalability and computational efficiency limitations in existing swarm coordination by tracking a single neighbor based on optic flow, achieving group cohesion and consensus.
Patent Information
- Application Number
- PCT/US2026/027137
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-07
- Filing Date
- 2026-05-07
- Publication Date
- 2026-11-12
AI Technical Summary
Existing swarm coordination approaches for autonomous agents require global position information, inter-agent communication, or dense connectivity, limiting their scalability and computational efficiency.
A bio-inspired visual processing system that enables motion coordination among mobile agents using passive sensing and computation, allowing each agent to identify and track a single neighboring agent based on optic flow cues, establishing a sparse network without explicit communication links, and implementing a 'scan, max, track' algorithm for motion synchronization.
Achieves scalable motion coordination with low computational complexity, enabling group cohesion and instantaneous consensus among large groups of agents, mimicking natural swarm behavior without strict convergence to a rigid formation.
Smart Images

Figure US2026027137_12112026_PF_FP_ABST
Abstract
Description
ELECTRONICALLY TRANSMITTED: MAY 7, 2026 PATENT SYSTEM AND METHOD FOR BIO-INSPIRED VISUALLY-GUIDED MULTI-AGENT COORDINATION REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to and benefit of U. S. provisional application Ser. No.63 / 801,548 filed May 7, 2025, which is hereby incorporated by reference in its entirety.STATEMENT AS TO RIGHTS TO INVENTIONS MADE UNDER FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT
[0002] Not Applicable.BACKGROUND ART
[0003] Many definitions of swarm and flock behaviors include three basic components: flock centering, velocity alignment among neighboring agents, and collision avoidance. These components support aggregation, velocity alignment, and obstacle avoidance as seen in group behavior of animals.
[0004] Various analytical and computational models have been proposed to describe swarming behavior, which can be broadly classified into the following: spatial which takes into account the surrounding space or environment either explicitly or implicitly; and topological or graph theoretic approaches which depend on the inherent group dynamics such as the connectivity and size of the swarm. In parallel physicists have developed theoretic models of group behavior mostly by modeling the mobile agents as particles, assigning interaction rules and studying the collective behavior emerging from their interactions.
[0005] Research on biological swarm behavior has identified the influence of a minority of informed individuals to guide the rest of the group and mathematical models have also been developed to show the importance of the number of informed individuals and swarm size to achieve group consensus. Mathematical models have been developed to analyze how the number of informed individuals, group size, and interaction rules affect collective decision-making and motion coherence.
[0006] Over the last decade, considerable research on individual insect flight modeling and its validation on robots has targeted their individual sensing and feedback strategies, from using system identification to extract flight dynamics models to implementation in autopilots, altitude estimation, landing maneuvers, navigation and collision detection.
[0007] Optic flow as visual feedback has been used in obstacle avoidance and group formation control in unmapped environments. Engineered approaches to computer-vision based feedback often use sparse feature selection from surroundings to perform visual odometry, pose estimation and SLAM (simultaneous location and mapping) in computationally parsimonious systems, motivated by analogy to insects using feature tracking for navigation and target detection. Despite the considerable focus on developing algorithms for robust feature tracking and localization, achieving biological efficiency in terms of computational speed and performance to parse an entire surrounding in limited time is still a challenge.
[0008] A navigation framework based on the insect visual motion sensitivity model that has growing experimental support is wide-field integration (WFI), in which an operator-theoretic formulation is implemented as the inner product of the optic flow with a pre-defined set of basis functions, an operation which generates an estimate of the change in relative velocity, orientation and position in terms of motion sensitivity patterns inspired from the spatial decomposition of optic flow by the motion sensitive wide-field neurons in the lobula complex of the visuomotor system of insects. While optic flow patterns model the visual system, wide-field integration models the neural system that analyses these patterns for motor response in biological insects.
[0009] Accordingly, there remains a need for multi-agent coordination frameworks that leverage passive sensing, including visual motion cues, to enable collective behavior without requiring global position information, inter-agent communication, or dense connectivity. What is needed is a system grounded in biologically inspired visual processing, and capable of operating with minimal sensing and computation on a per agent basis to address the limitations present in existing swarm coordination approaches, particularly when autonomous agents are involved.SUMMARY OF THE INVENTION
[0010] The present disclosure examines a well-developed insect visuomotor feedback model for individual insects, and extends the feedback model for a dynamic environment of multiagent group motion.
[0011] A system and method for managing a group of mobile autonomous agents is disclosed.
[0012] Described herein is a system that supports motion coordination that can be scaled to large scale groups of mobile agents (ground, aerial, and underwater). In some embodiments, the system supports the motion coordination without absolute position reference and with no explicit communication link, e.g., a radio frequency communication link, between the mobile agents. The system may include control logic comprising an algorithm stored on one or more non-transitory computer readable medium (e.g., RAM, ROM, flash memory or the like) that is configured to provide a computationally efficient way to spread the sensing, computation, and communication loads which are otherwise unscalable with large groups. The algorithm may do so by allowing each mobile agent to identify a single other mobile agent to regard. The demonstrated medium for this may be optical, representing an implicit visual communication pathway. By establishing such a link, a sparse network (relative to those used in traditional engineered motion synchronization mechanisms) is established, and the underlying graph network is never fully connected; (ie, it is always instantaneously disconnected). By appropriate choice of which mobile agent to select, the underlying leader -follower pairs achieve a motion synchronization.
[0013] The resulting motion is not strict convergence to a rigid formation, as seen in previous engineered approaches, but is a looser less polarized group that achieves a less easily predicted group.
[0014] For the ground and aerial robotics, the demonstrated implementation embodiment uses a panoramic camera view and computes a local spatial gradient across the image, also known as optic flow. The algorithm in the demonstrated implementation embodiment may be thought of as continuously implementing a "scan, max, track" at instants of time within a period of time.
[0015] Scan: look for a specific feature in the local spatial gradient, such as the maximum optic flow magnitude in both azimuth and elevation.
[0016] Max: identify the location of this maximum (azimuth, elevation). The mobile agent selects as a target a neighboring mobile agent producing the maximum optic flow magnitude within the visual field.
[0017] Track: pursue this mobile agent position, using a generalized pursuit / tracking rule. In examples described herein six motion coordination rules are demonstrated (including pure pursuit guidance laws, proportional navigation, motion camouflage guidance laws).
[0018] Other specific features in the local spatial gradient can also be identified and used to identify the location of the neighboring mobile agent by analyzing motion blur. Exemplary specific features identified by analyzing motion blur include a peak motion blur magnitude, a peak motion blur minimum magnitude, a blur inflection point, ora blurflat point. The present disclosure adds to the art providing bio-inspired multi-agent feedback control frameworks that overcome key limitations of prior approaches. Unlike conventional swarm algorithms that rely on communication, high-bandwidth state sharing, or tracking of multiple neighbors, the disclosed systems utilize passive visual sensing and achieve scalable small target motion detector (STMD) neurons as maxima and argument of maxima operator on optic flow. The disclosure enables each mobile agent to target a single neighboring mobile agent at any instant:, e.g., the neighboring mobile agent who generates the highest optic flow magnitude. The bearing angle to said neighboring mobile agent is then used as direct feedback for motion control. This selective attention mechanism, combined with a switched feedback framework (governed by dwell time constraints) ensures bounded mobile agent trajectories without requiring asymptotic convergence. This creates motion that more closely mimics the behavior of insects in natural swarms.
[0019] The disclosed approach achieves group cohesion and instantaneous consensus while each mobile agent is connected to (e.g., monitoring relative position of) as few as one neighboring mobile agent, and while spreading attention work over time resulting in a constant per-agent computational workload corresponding to the lowest possible order of computational complexity (0(1) complexity) per mobile agent. The framework is platform agnostic, and has been reduced to practice on mobile agents including ground robots, aerial vehicles, and in simulations with space craft, demonstrating robust performance in communications-denied and GPS denied environments.
[0020] The foregoing Summary provides an overview of certain selected implementations or embodiments disclosed herein, and is not intended to describe every aspect, embodiment, implementation, feature, or advantage of the disclosure exhaustively or comprehensively. Therefore, this Summary should not be construed in such a way to limit the scope of this disclosure or to limit the scope of the claims. The details of one or more implementation or embodiment disclosed herein are set forth in the accompanying drawings and descriptions below. Other aspects, features, implementations, embodiments, and advantages will become readily apparent in view of the description, the drawings, and the claims set forth herein.BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more implementations described herein and, together with the description, explain these implementations. Not every component may be labeled in every drawing. Like reference numerals in the figures may represent and refer to the same or similar element or function. In the drawings:
[0022] FIG. 1 is a diagram showing an exemplary group of mobile agents in an exemplary formation.
[0023] FIG. 2 is a block diagram showing an exemplary mobile agent constructed in accordance with the present disclosure.
[0024] FIG. 3 is a block diagram illustrating an example visual data structure transformation pipeline for a mobile agent.
[0025] FIG.4 illustrates a geometry of optic flow definition according to one embodiment of the disclosure, showing the relationship between an observer and a target in a spherical coordinate system.
[0026] FIG. 5 illustrates geometric parameters of a bio-inspired agent operating within an idealized group of mobile agents in a formation according to one embodiment.
[0027] FIG. 6A is a graph illustrating an exemplary motion sensitivity pattern of the vertical motion detecting (fourth neuron " VS4") tangential cells in the lobula plate section of an insect visuomotor sensor system.
[0028] FIG. 6B is a graph illustrating an exemplary motion sensitivity pattern of the horizontal motion detecting tangential cells in a lobula plate section of the insect visuomotor sensor system.
[0029] FIG. 7 is a block diagram of a bio-inspired wide-field integration (WFI) based closed-loop feedback control system.
[0030] FIG. 8A is a graph illustrating piecewise optic flow pattern with some relative velocity at initial conditions of an agent.
[0031] FIG. 8B is a graph illustrating piecewise optic flow pattern for relative velocity change of an agent.
[0032] FIG. 8C is a graph illustrating piecewise optic flow pattern for lateral position change of an agent.
[0033] FIG. 8D is a graph illustrating piecewise optic flow pattern for orientation change of an agent.
[0034] FIG. 9A is a graph illustrating agent trajectories when introduced to simulation generated by a traditional Cucker-Smale model before convergence to eventual group velocity.
[0035] FIG. 9B is a graph illustrating agent trajectories when introduced to simulation generated by a traditional Cucker-Smale bounded distance between agents.
[0036] FIG. 9C is a graph illustrating agent trajectories when introduced to simulation generated by a traditional Cucker-Smale model motion trajectories of all agents.
[0037] FIG. 10A is a graph illustrating change in orientation offset over time in bio-inspired agent simulation where orientation regulation of bio-inspired agent, from an initial offset of 0 = 45 °, using idealized optic flow computation for wide-field integration feedback in a group of 50 swarm agents.
[0038] FIG. 10B is a graph illustrating bounded distance between agents over time in a bioinspired agent simulation where orientation regulation of bio-inspired agent, from an initial offset of 0 = 45 °, using idealized optic flow computation for wide-field integration feedback in a group of 50 swarm agents.
[0039] FIG. 10C is a graph motion trajectories of mobile agents in a bio-inspired agent simulation where orientation regulation of bio-inspired mobile agent, from an initial offset of 0 = 45 °, using idealized optic flow computation for wide-field integration feedback in a group of 50 swarm mobile agents.
[0040] FIG. 11 is a top view of a testbed for a swarm of Robotarium mobile agents. (Dissertation Figure 11.8)
[0041] FIG. 12A is a graph illustrating mean theta in a simulation result where bio inspired agent velocity is less than that of swarm agents.
[0042] FIG. 12B is a graph illustrating angular velocity in a simulation result where bio inspired agent velocity is less than that of swarm agents.
[0043] FIG. 12C is a graph illustrating relative motion of mobile agents in swarm along with variance in a simulation result where bio inspired agent velocity is less than that of swarm agents.
[0044] FIG. 13 is a graph illustrating time history of the planar optic flow components produced by each of the swarming agents as experienced by the bio-agent during oneexperimental run of the first case where velocity of bio-inspired agent is less than that of swarm.
[0045] FIG. 14A is a graph illustrating mean theta in a simulation result where bio inspired agent velocity is equal to that of swarm agents.
[0046] FIG. 14B is a graph illustrating angular velocity time history in a simulation result where bio inspired agent velocity equal to that of swarm agents.
[0047] FIG. 14C is a graph illustrating relative motion of mobile agents in swarm along with variance in a simulation result where bio inspired agent velocity is equal to that of swarm agents.
[0048] FIG. 15 is a graph illustrating time history of the planar optic flow components produced by each of the swarming agents as experienced by the bio-agent during one experimental run of the first case where velocity of bio-inspired agent is equal to that of swarm.
[0049] FIG. 16A illustrates trajectories of five agents in simulation all implementing parallelformation controller with collision avoidance as taught by Nima Moshtagh, Nathan Michael, Ali Jadbabaie, and Kostas Daniilidis, Vision-based, distributed control laws for motion coordination of nonholonomic robots, IEEE Transactions on Robotics 25 (2009), no. 4, 851-860 (hereinafter referred to as Moshtagh et al.).
[0050] FIG. 16B illustrates Simulation results for an agent separately applying the idealized proportional controller, the multi-agent feedback controller developed in accordance with the present disclosure (Eqn. 2.1.5) and the feedback controller proposed in Moshtagh et al. for parallel-formation, without collision avoidance, for an established swarm formation and arbitrary initial condition.
[0051] FIG. 17A is a block diagram of a system with uncertainties according to one embodiment.
[0052] FIG. 17B is a block diagram of a generalized model of system uncertainty according to one embodiment.
[0053] FIG. 18 is an illustration of time varying formation configurations illustrating previously unmodeled geometric configuration uncertainty.
[0054] FIG. 19A illustrates an embodiment of an agent constructed in accordance with the present disclosure in which the mobile agent is constructed for traversing a solid surface, and includes a camera sensor.
[0055] FIG. 19B is an illustration of the mobile agent of FIG. 19A having a marker for optical recognition.
[0056] FIG. 19C is an illustration of the experimental testbed setup having the mobile agents depicted in FIGs 19Aand 19B.
[0057] FIG.20A illustrates a first stage in optic flow processing from a 360° camera, including a raw camera image.
[0058] FIG. 20B illustrates a distortion-corrected image from a front-facing 360 degree camera.
[0059] FIG.20C illustrates optic flow measured for fixed discrete points along one horizontal line of a de-warped image.
[0060] FIG.21Aisagraph illustrating singular values of a loop transfer function in Eqn. (2.2.8) with ground vehicle kinematics as plant model
[0061] FIG. 21B is a graph illustrating singular values of insect dynamics from Eqn. (2.2.5) as plant model.
[0062] FIG. 22A is a singular value plot of the weight transfer functions.
[0063] FIG. 22B is a singular value plot of weighted output complementary sensitivity function.
[0064] FIG. 22C is a singular value plot of lower arbitrary gains K and Wofrom Eqn. (2.2.11).
[0065] FIG. 23A is a graph illustrating mean orientation experimental results for 20 trials of the first case with some gain KltK2<0 and 0O= 45.
[0066] FIG.23B is a graph illustrating mean angular velocity experimental results for 20 trials of the first case with some gain KltK2< 0 and 0O= 45.
[0067] FIG.23C is a graph illustrating mean agent trajectory experimental results for 20 trials of the first case with some gain KltK2< 0 and 0O= 45.
[0068] FIG. 24 is a graph of experimental results containing sensor noise vs nominal (agent) results illustrate the effects of sensor noise.
[0069] FIG. 25A is a graph of experimental results for different initial angular offsets: mean orientation.
[0070] FIG. 25B is a graph of experimental results for different initial angular offsets: mean trajectories.
[0071] FIG. 26A is a graph of experimental results for 20 trials of the second case with lower gains: mean orientation.
[0072] FIG. 26B graph of experimental results for 20 trials of the second case with lower gains: mean angular velocity.
[0073] FIG. 26C graph of experimental results for 20 trials of the second case with lower gains: mean trajectory of the bio agent.
[0074] FIG. 27 illustrates the geometry of body-relative head motion in the presence of a target according to one embodiment of an optic flow enrichment system.
[0075] FIG. 28 is a block diagram of a closed-loop feedback model for optic flow enrichment according to one embodiment.
[0076] FIGS. 29A-H illustrates a time-domain response of a bio-inspired agent operating under optic-flow-based feedback with motion enrichment. Together, these responses characterize the closed-loop behavior of the mobile agent as it regulates orientation, velocity, and position using visually derived feedback signals.
[0077] FIG. 29A is a time history of bio-agent forward speed, Vb.
[0078] FIG. 29B is a time history of relative range between the bio-agent and neighboring agent, p.
[0079] FIG. 29C is a time history of bio-agent body roll angle, 0.
[0080] FIG. 29D is a time history of bio-agent yaw rate, r.
[0081] FIG. 29E is a time history of bio-agent heading angle, 0b.
[0082] FIG. 29F is a time history of bio-agent lateral position, yb.
[0083] FIG. 29G is a time history of bio-agent control input, ub.
[0084] FIG. 29H is a time history of bio-agent longitudinal position change, xb.
[0085] FIG. 30 is a block diagram of a small-target motion reactive (STMR) bio-inspired multi-agent closed-loop feedback model based on STMD neurons as a sensory system according to one embodiment.
[0086] FIG. 31 is a mathematical illustration of the geometry of a bi-agent problem.
[0087] FIG. 32{A— F) illustrate simulation results for a multi-agent switched feedback system based on optic-flow sensing, showing agent trajectories under six different initial spatial configurations. In the simulation, a plurality of mobile agents move in a planar environment while selectively switches attention among neighboring agents based on optic-flow magnitude, subject to an average dwell time constraint that limits the rate of switching.
[0088] FIG.32A illustrates a first initial condition in which the plurality of mobile agents begin in a relatively compact configuration near the origin with differing initial headings. The trajectories illustrate the evolution of mobile agent motion underthe switched feedback law.
[0089] FIG. 32B illustrates a second initial condition in which the mobile agents are initially distributed along a horizontal axis, producing different optic-flow interactions among neighboring agents as the system evolves.
[0090] FIG. 32C illustrates a third initial condition in which the mobile agents begin in a clustered configuration offset within the planar environment.
[0091] FIG. 32D illustrates a fourth initial condition in which the mobile agents begin in an elongated spatial distribution, increasing the variation in inter-agent optic-flow measurements during the initial phase of the simulation.
[0092] FIG. 32E illustrates a fifth initial condition in which the mobile agents begin in a wider spatial configuration with larger inter-agent spacing.
[0093] FIG. 32F illustrates a sixth initial condition in which the mobile agents begin in a dispersed configuration across the planar region. In each instance, the trajectories illustrate the resulting collective motion of the mobile agents when operating under the disclosed switched feedback control framework.
[0094] FIG. 33{A— F) illustrate graphs of a time evolution of mobile agent orientation corresponding to the six initial conditions shown in FIG. 32{A-F). Each graph shows the orientation of a plurality of mobile agents as a function of time while the mobile agents operate underthe same optic-flow-driven switched feedback system with average dwell time constraints.
[0095] FIG. 33A illustrates the orientation trajectories for the mobile agents corresponding to the initial configuration shown in FIG. 32A.
[0096] FIG. 33B illustrates the orientation trajectories corresponding to the initial configuration shown in FIG. 32B.
[0097] FIG. 33C illustrates the orientation trajectories corresponding to the initial configuration shown in FIG. 32C.
[0098] FIG. 33D illustrates the orientation trajectories corresponding to the initial configuration shown in FIG. 32D.
[0099] FIG. 33E illustrates the orientation trajectories corresponding to the initial configuration shown in FIG. 32E.
[0100] FIG. 33F illustrates the orientation trajectories corresponding to the initial configuration shown in FIG. 32F. In each case, the plots illustrate the temporal response of the multi-agent system as the mobile agents adjust headings based on the optic-flow-driven switching mechanism.
[0101] FIG. 34A illustrates simulation results of switching signal plot showing the mobile agents "tracked" over time by a single agent (fifth agent from top in FIG. 32E) in the multiagent switched system with and without the dwell time constraints, and the average dwell time plot.
[0102] FIG. 34B illustrates simulation results for the same agent in 34A with the minimum average dwell time plotted.
[0103] FIG. 35(A— F) illustrate simulation results for a multi-agent switched feedback system based on optic-flow sensing, showing agent trajectories under six different initial spatial configurations when the switching of target agents is performed without an average dwell time constraint. In the simulation, a plurality of mobile agents move in a planar environment while each mobile agent selectively switches attention among neighboring agents based on optic-flow magnitude.
[0104] FIG.35A illustrates a first initial condition in which the plurality of mobile agents begin in a relatively compact configuration near the origin with differing initial headings. The trajectories illustrate the evolution of mobile agent motion under the switched feedback law without dwell time constraints.
[0105] FIG. 35B illustrates a second initial condition in which the mobile agents are initially distributed along a horizontal axis, producing different optic-flow interactions among neighboring agents as the system evolves.
[0106] FIG. 35C illustrates a third initial condition in which the mobile agents begin in a clustered configuration offset within the planar environment.
[0107] FIG. 35D illustrates a fourth initial condition in which the mobile agents begin in an elongated spatial distribution, increasing the variation in inter-agent optic-flow measurements during the initial phase of the simulation.
[0108] FIG. 35E illustrates a fifth initial condition in which the mobile agents begin in a wider spatial configuration with larger inter-agent spacing.
[0109] FIG. 35F illustrates a sixth initial condition in which the mobile agents begin in a dispersed configuration across the planar region. In each instance, the trajectories illustratethe resulting collective motion of the mobile agents when operating under the disclosed switched feedback control framework without the dwell time constraint.
[0110] FIG. 36A a graph illustrating experimental results for the implementation of multiagent STMR feedback control system with two different feedback rules considered: pursuit feedback rule from Eqn. 3.3.4.
[0111] FIG. 36B a graph illustrating experimental results for the implementation of multiagent STMR feedback control system with two different feedback rules considered: motion camouflage.
[0112] FIG. 37is a graph of a single feedback-controlled mobile agent surrounded by non-reactive neighbors with established formation; comparison of STMR feedback control system with the optomotor feedback control system from, the Vicsek model and the Cucker-Smale model.
[0113] FIG. 38A Graph illustrating a collection of feedback-controlled agents all reactive to each other.
[0114] FIG. 38B is a graph illustrating a comparison of STM feedback controls with the optomotor feedback control system from the Vicsek model.
[0115] FIG. 38C is a graph illustrating a comparison of STM feedback controls with the optomotor feedback control system from the C-S model.
[0116] FIG. 39A is a graph illustrating comparison of algebraic connectivity (Fiedler eigenvalue) among the STMR feedback control system proposed within the present disclosure, the optomotor feedback control system from the Vicsek model
[0117] FIG. 39B is a graph illustrating comparison of algebraic connectivity (Fiedler eigenvalue) among the STMR feedback control system proposed within the present disclosure, the optomotor feedback control system from the Cucker-Smale model.
[0118] FIG. 40 graph of simulation results for homogeneous agents following the STMR feedback framework.
[0119] FIG.41(A— F) illustrate simulation results for a multi-agent optic-flow-based feedback system showing trajectories of a plurality of mobile agents under six different initial spatial configurations. In the simulation, each mobile agent determines control inputs based on visual motion cues derived from surrounding agents and operates under the disclosed optic-flow-based feedback framework.
[0120] FIG.41A illustrates a first initial condition in which the plurality of mobile agents begin in a relatively compact configuration near the origin with differing initial headings. The trajectories illustrate the resulting motion of the mobile agents under the optic-flow-based feedback control system.
[0121] FIG. 41B illustrates a second initial condition in which the mobile agents are initially distributed along a horizontal axis, producing varying optic-flow interactions among neighboring agents as the system evolves.
[0122] FIG. 41C illustrates a third initial condition in which the mobile agents begin in a clustered configuration offset within the planar environment.
[0123] FIG. 41D illustrates a fourth initial condition in which the mobile agents begin in an elongated spatial distribution, increasing variation in inter-agent visual motion cues during the initial phase of the simulation.
[0124] FIG.41E illustrates a fifth initial condition in which the mobile agents begin in a wider spatial configuration with larger inter-agent spacing.
[0125] FIG. 41F illustrates a sixth initial condition in which the mobile agents begin in a dispersed configuration across the planar region. In each instance, the trajectories illustrate the resulting collective motion of the mobile agents when operating under the disclosed optic-flow-based feedback control framework.
[0126] FIG. 42A is a graph illustrating average orientation of swarm agents and streaker agents during the STMR simulation for two instances of initial conditions in FIG.41C.
[0127] FIG. 42B is a graph illustrating average orientation of swarm agents and streaker agents during the STMR simulation for two instances of initial conditions in FIG.41F.
[0128] FIG.43 is a graph illustrating convergence results from experiments graphed in FIGS.42A and 42B.
[0129] FIG.44{A— F) illustrate simulation results for a multi-agent optic-flow-based feedback system showing trajectories of a plurality of mobile agents under six different initial spatial configurations in the presence and absence of higher-speed "streaker" agents. In the simulation, a subset of mobile agents move along fixed trajectories at higher speeds while the remaining agents operate under the disclosed optic-flow-based feedback framework, allowing the influence of the streaker agents on swarm motion to be observed.
[0130] FIG.44A illustrates a first initial condition in which the plurality of mobile agents begin in a relatively compact configuration near the origin with differing initial headings, showingtrajectories of the swarm agents with and without the presence of the higher-speed streaker agents.
[0131] FIG. 44B illustrates a second initial condition in which the mobile agents are initially distributed along a horizontal axis, demonstrating the resulting swarm trajectories under the optic-flow-based feedback system with and without streaker agents.
[0132] FIG. 44C illustrates a third initial condition in which the mobile agents begin in a clustered configuration offset within the planar environment, illustrating the influence of the streaker agents on the resulting swarm trajectories.
[0133] FIG. 44D illustrates a fourth initial condition in which the mobile agents begin in an elongated spatial distribution, increasing variation in inter-agent visual motion cues and showing the effect of the streaker agents on swarm motion.
[0134] FIG.44E illustrates a fifth initial condition in which the mobile agents begin in a wider spatial configuration with larger inter-agent spacing, demonstrating how the swarm trajectories respond to the presence of the streaker agents.
[0135] FIG. 44F illustrates a sixth initial condition in which the mobile agents begin in a dispersed configuration across the planar region, illustrating the resulting swarm motion both with and without the higher-speed streaker agents.
[0136] FIG. 45A is a graph illustrating average orientation of swarm agents and streaker agents during the MA-WFI simulation for two instances of initial conditions in FIG.44C.
[0137] FIG. 45B is a graph illustrating average orientation of swarm agents and streaker agents during the MA-WFI simulation for two instances of initial conditions in FIG.44F.
[0138] FIG. 46A is a graph illustrating trajectories of swarm agents and streaker agents during the STMR simulation for a first instance of initial conditions.
[0139] FIG. 46B is a graph illustrating trajectories of swarm agents and streaker agents during the STMR simulation for a second instance of initial conditions.
[0140] FIG.47 is a graph illustrating average swarm orientation results corresponding to the simulation instances shown in FIGS.46A and 46B.
[0141] FIG.48A is an illustration of a STMR aircraft setup with a 360 camera on a canopy.
[0142] FIG. 48B is an illustration of the components inside the airframe from the STMR aircraft setup in 48A.
[0143] FIG. 49 is a block diagram illustrating an STMR feedback loop for the aircraft experiment.
[0144] FIG. 50A is an illustration of planned trajectories of the airframe and stimulus for flight test implementation.
[0145] FIG. 50B is an illustration of a pneumatic launcher for a projectile stimulus.
[0146] FIG. 51 illustrates camera sensor output and optic flow calculation.
[0147] FIG. 52 illustrates flight trajectories for the stimulus and the STMR aircraft for three different runs.
[0148] FIG. 53A illustrates simulation results for a space-based implementation of the optic-flow-based multi-agent control framework showing relative motion trajectories of mobile agents operating within a Hill's reference frame during a rendezvous scenario.
[0149] FIG. 53B illustrates corresponding simulation results for the space-based implementation of FIG. 53A, showing additional characteristics of the relative motion behavior of the mobile agents under the disclosed feedback control system.
[0150] FIG. 54 is a diagram illustrating Hill's reference frame for relative motion of satellites according to a space-based embodiment.
[0151] FIG. 55A is a graph illustrating orbit of satellite and a target in an Earth-centered inertial frame without the optic flow feedback controller.
[0152] FIG. 55B is a graph illustrating orbit of satellite and a target in an Earth-centered inertial frame with the optic flow feedback controller.
[0153] FIG. 55A is a graph illustrating a three-dimensional trajectory of a chief satellite approaching a target in an Earth-centered inertial frame without optic feedback controller.
[0154] FIG. 56A is a graph illustrating a three-dimensional trajectory of a satellite approaching a target in an Earth centered inertial frame without an optic feedback controller.
[0155] FIG. 56B is a graph illustrating a three-dimensional trajectory of a satellite approaching a target in an Earth-centered inertial frame with optic feedback controller.
[0156] FIG. 57A is a graph illustrating the x, y, and z components of relative velocity during an orbital rendezvous without the (optic feedback) controller.)
[0157] FIG. 57B is a graph illustrating the x, y, and z components of relative velocity during an orbital rendezvous with the (optic feedback) controller.
[0158] FIG. 58A is a graph illustrating relative velocity of the satellite and target without the controller.
[0159] FIG. 58B is a graph illustrating relative velocity of the satellite and target with the controller.
[0160] FIG. 59A is a graph illustrating relative distance between satellites rendered in a rotating reference frame without control feedback.
[0161] FIG. 59B is a graph illustrating relative distance between satellites rendered in a rotating reference frame with control feedback.
[0162] FIG. 60A is a graph illustrating a three-dimensional relative position (p) of the satellite without the controller.
[0163] FIG. 60B is a graph illustrating a three-dimensional relative position (p) of the satellite with the controller.
[0164] FIG. 61A is a graph illustrating a three-dimensional relative velocity (p) of the satellite without the controller.
[0165] FIG. 61B is a graph illustrating a three-dimensional relative velocity (p) of the satellite with the controller.DETAILED DESCRIPTION
[0166] The following detailed description refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.
[0167] Terms Defined:
[0168] As used herein, the terms "mobile agent", "bio-agent", "bio-inspired agent", "streaker agent", "swarm agent", "target agent", or simply "agent" refers to a mobile platform capable of autonomous or semi-autonomous motion, including but not limited to ground vehicles, unmanned aerial vehicles (UAVs), spacecraft, marine vessels, and underwater vehicles. A "mobile agent", "bio-agent", "bio-inspired agent", "streaker agent", "target agent", "swarm agent", or simply "agent" designed in accordance with the present disclosure may include sensing hardware, onboard processing hardware, control logic, and actuation mechanisms configured to generate motion in response to visual information obtained from one or more neighboring agent. The "target neighboring agent" or "neighboring agent" in accordance with the present disclosure is the neighboring agent selected based on the optic flow magnitude criterion.
[0169] The term "optic flow" refers to a pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and the scene. The geometry of optic flow definition is illustrated in FIG.4, showing the relationship between an observer, a target, and a spherical coordinate system.
[0170] The term "information flow" refers to the propagation of motion influence through the swarm via visual sensing relationships.
[0171] The term "wide-field integration" refers to a mathematical operation that extracts motion cues from optic flow by computing spatial inner products with predetermined basis functions.
[0172] The term "convergent boundedness" as used herein, refers to a stability property of the closed-loop multi-agent feedback system in which the dynamic states of each mobile agent, including position, orientation, and velocity, remain bounded over time while the mobile agents adjust their motion according to the disclosed optic-flow-based feedback laws. In particular, convergence boundedness means that the trajectories of the mobile agents do not diverge to infinity and remain confined within a finite region of the state space while the mobile agents approach coordinated motion. A "theoretical convergence analysis" is a mathematical demonstration that agent states governed by a proposed control law evolve toward and remain within a bounded neighborhood of a desired coordinated configuration overtime.
[0173] The term "small-target motion detector" or " STMD" refers to a neural mechanism, or engineered analog thereof, that responds preferentially to at least one moving neighboring agent in a visual field.
[0174] The term "graph-theoretic representation" refers to a way of modeling a multi-agent system (like a swarm) as a mathematical graph (possibly comprising nodes and edges).
[0175] Examples of a mobile agent, as used herein include, but are not limited to, a robot, an unmanned aerial vehicle, a submarine, an unmanned surface vessel, a helicopter or combinations thereof.
[0176] Examples of a sensory system, as used herein include, but are not limited to, an optical detector (camera or video camera), a radar, a lidar or combinations thereof.
[0177] Examples of visual information as used herein include, but are not limited to, an image or a series of images, a depth map, or a series of depth maps, reflectivity data, or a series of reflectivity data, or the like.
[0178] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listedor inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or and not to an exclusive or. Forexample, a condition A or B is satisfied by anyone of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
[0179] In addition, use of the "a" or "an" are employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the inventive concept. This description should be read to include one or more and the singular also includes the plural unless it is obvious that it is meant otherwise.
[0180] Further, use of the term "plurality" is meant to convey "more than one" unless expressly stated to the contrary.
[0181] As used herein, qualifiers like "substantially," "about," "approximately," and combinations and variations thereof, are intended to include not only the exact amount or value that they qualify, but also some slight deviations therefrom, which may be due to manufacturing tolerances, measurement error, wear and tear, stresses exerted on various parts, and combinations thereof, for example.
[0182] The use of the term "at least one" or "one or more" will be understood to include one as well as any quantity more than one. In addition, the use of the phrase "at least one of X, V, and Z" will be understood to include X alone, V alone, and Z alone, as well as any combination of X, V, and Z.
[0183] The use of ordinal number terminology (i.e., "first", "second", "third", "fourth", etc.) is solely for the purpose of differentiating between two or more items and, unless explicitly stated otherwise, is not meant to imply any sequence or order or importance to one item over another or any order of addition.
[0184] Where a rangeof numerical valuesis recited or established herein, the range includes the endpoints thereof and all the individual integers and fractions within the range, and also includes each of the narrower ranges therein formed by all the various possible combinations of those endpoints and internal integers and fractions to form subgroups of the larger group of values within the stated range to the same extent as if each of those narrower ranges was explicitly recited. Where a range of numerical values is stated herein as being greater than a stated value, the range is nevertheless finite and is bounded on its upper end by a value that is operable within the context of the disclosure as described herein. Where a range ofnumerical values is stated herein as being less than a stated value, the range is nevertheless bounded on its lower end by a non-zero value. It is not intended that the scope of the disclosure be limited to the specific values recited when defining a range. All ranges are inclusive and combinable.
[0185] As used herein, any reference to "one embodiment," "an embodiment," "some embodiments," "one example," "for example," or "an example" means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment and may be used in conjunction with other embodiments. The appearance of the phrase "in some embodiments" or "one example" in various places in the specification is not necessarily all referring to the same embodiment, for example.
[0186] The various algorithms described herein can be implemented with circuitry. As used herein, "circuitry" may refer to analog and / or digital components, or one or more suitably programmed processors (e.g., microprocessors) and associated hardware and software, or hardwired logic. Also, "circuitry" may perform one or more functions. The term "circuitry" may include hardware, such as a processor (e.g., microprocessor), a combination of hardware and software, and / or the like. Software may include one or more processor-executable instructions that when executed by one or more processors cause the one or more processors to perform a specified function.
[0187] As used herein, "software" may include one or more computer readable instruction that when executed by one or more component (e.g., a processor) causes the component to perform a specified function. It should be understood that the algorithms described herein may be stored on one or more non-transitory computer-readable medium. Exemplary non-transitory computer-readable media may include a non-volatile memory, a volatile memory, a random-access memory (RAM), a read only memory (ROM), a CD-ROM, a hard drive, a solid-state drive, a flash drive, a memory card, a DVD-ROM, a Blu-ray Disk, a laser disk, a magnetic disk, an optical drive, a phase change memory, combinations thereof, and / or the like. Such non-transitory computer-readable media may be electrically based, optically based, magnetically based, material-phase based, resistive based, and / or the like. Further, the messages described herein may be generated by the components and result in various physical transformations.
[0188] The Following legend is also included for definition purposes.WFI wide-field integrationMA — WFI multi-agent wide-field integrationSTMD small-target motion detector neuronsSTMR small-target motion reactiveQ optic flowbio-agent statesqtneighboring agent states0bbio-agent heading angle0tneighboring agent heading anglexbbio-agent velocity along x-axisxtneighboring agent velocity along x-axisybbio-agent velocity along y-axisytneighboring agent velocity along y-axisY bearing angleI. Overview
[0189] In some embodiments, the present disclosure provides bio-inspired multi-agent feedback control frameworks that enable autonomous agents to coordinate their motion using only passive onboard optical sensing. The disclosure is motivated by the observation that flying insects achieve robust group coordination using relatively small collections of neurons, relying primarily on visual cues such as optic flow: the pattern of apparent motion of objects across the retina.
[0190] FIG. 1 illustrates an exemplary formation 90 of mobile agents lOOa-lOOe in an exemplary formation. Five of the mobile agents 100 are provided in FIG. 1 by way of example, although more or less of the mobile agents 100 may be provided in a formation for coordinated movement as described herein. All of the mobile agents 100 may be similar in construction and function, except as described herein. One or more mobile agents 100 mayact as an informed agent, such as the mobile agent 100 configured as the informed agent 105 shown in FIG. 1, which is also referred to hereafter as a "streaker agent." One or more of the mobile agents 100 that may be influenced by the informed agent 105 are referred to herein as an uninformed agent and are labeled in FIG. 1 with the reference numeral 110a, 110b, 110c and llOd. The mobile agents 100 including the informed agent 105 and the uninformed agents llOa-d are spaced apart and move in a coordinated manner as discussed herein. More particularly, the informed agent 105 may navigate dynamically or in a predetermined manner without reference to any other mobile agent 100. The informed agent 105 possess knowledge of a mission / destination goal and direct the uninformed agent(s) llOa-d by injecting motion / momentum into the swarm. The other mobile agents 100 configured as uninformed agents llOa-d, however, are without knowledge of mission goals, and coordinate motion by monitoring location, orientation and / or movement of a neighboring mobile agent 100 as described herein. In some embodiments, the uninformed agents llOa-d rely on STMR (Small Target Motion Reactivity) rules described herein to maintain cohesion without knowledge of the mission goals.
[0191] For example, for the uniformed mobile agent 110a, the informed agent 105 and the uninformed mobile agents llOb-d may be neighboring mobile agents. Similarly, for the uniformed mobile agent 110b, the informed agent 105 and the uninformed mobile agents llOa-d may be neighboring mobile agents.
[0192] Shown in FIG. 2 is a block diagram of one of the mobile agents 100. In operation, a plurality of mobile agents 100 form a multi-agent system in which each uninformed mobile agent 110 performs sensing and control independently based on locally observed visual information. Each uninformed mobile agent 110 identifies at least one neighboring mobile agent 100 (either the informed mobile agent 105 or the uninformed mobile agent 110) based on visual cues derived from optic flow. The informed mobile agent 105 navigates dynamically or in a predetermined manner without reference to any other mobile agent 100. The mobile agent 100 generally includes a sensor system 202, one or more actuator or output elements 204, one or more controller 206, one or more processors 208, one or more communication interfaces 212, and one or more non-transitory computer readable media 218 storing executable control logic which may be in the form of firmware or software. The computational model 218a represents the transformation from visual sensory data input into internal motion-related variable. The control logic 218b converts the outputs of the computationalmodel into actuator commands. The controller 206 includes circuitry to manage and direct data between the sensory system 202, the actuator 204 and the communication interfaces 212. In some embodiments, the controller 206 includes the one or more processors 208, and the non-transitory computer readable media 218.
[0193] The informed mobile agent(s) 105 and the uninformed mobile agent 110 may comprise the exact same hardware components depicted in FIG. 2 where distinct roles of the informed mobile agent 105 and the uninformed mobile agent 110 are dictated by programmed software instructions stored in the non-transitory computer readable media 218 ((e.g., executing STMR vs. executing a predefined trajectory).
[0194] For example, the mobile agent 100 when configured as the informed mobile agent 105 may not be provided with the computational model 218a or the control logic 218b. Rather, the mobile agent 100 when configured as the informed mobile agent 105 may include navigation logic to assist the informed mobile agent 105 in navigating dynamically. To be able to act as both the informed agent 105 and the uninformed agent 110, the sensor system 202 may also include circuitry for determining a three dimensional location of the mobile agent 100 and providing information regarding the three dimensional location of the mobile agent 100 to the processor 208 executing the navigation logic to cause the mobile agent 100 configured as the informed agent 105 to navigate.
[0195] In some embodiments, the informed mobile agent 105 can be constructed in a manner different from the uninformed mobile agent 110. For example, informed agents 105 may comprise specialized hardware (e.g., advanced communication transceivers, global positioning sensors and the like) to run mission campaigns. The uninformed agent 110, however, may not be provided with the communication transceivers, global positioning sensors and the like to provide independent navigation. In these embodiments, the processor 208 of the mobile agent 100 configured as the uninformed agent 110 makes decisions relying strictly on local visual information (e.g., a visual imprint / optic flow) from the sensory system 202. the processor 208 of the mobile agent 100 configured as the uninformed agent 110 may also not be provided with instructions to compute or possess the true global position, true orientation, or true velocity of any other mobile agent 100 in the swarm. Further, in some embodiments, the processor 208 of the mobile agent 100 configured as the uninformed agent 110 may also not be provided with instructions to compute or possess graph information withrespect to at least a portion ofthe swarm or compute actuator command(s) based upon graph information with respect to at least a portion ofthe swarm.
[0196] The following describes the construction and use of the mobile agent 100 configured as the uninformed agent 110. In these embodiments, the sensory system 202 includes circuitry configured to perceive visual information from at least one neighboring mobile agent 100 during a time period. The sensory system 202 may include optical sensors, cameras, inertial measurement units, proximity sensors, or other environmental sensing components configured to detect visual or motion cues of the neighboring mobile agent 100 in the environment ofthe mobile agent 100.
[0197] The actuator or output elements 204 may include circuitry, propulsion elements, wing actuation mechanisms, control surfaces, indicator devices, and / or other motiongenerating or signaling components capable of controlling motion of the mobile agent 100 in response to commands generated by the processor 208.
[0198] The processor 208 may include a microcontroller, system-on-chip processor, digital signal processor, or other embedded processing hardware configured to execute control instructions and process sensor data received from the sensory system 202. The sensory system 202 may include an optic flow sensor, camera, event-based vision sensor, or equivalent visual motion sensing device.
[0199] The communication interface 212 may provide wired or wireless communication, utilizing a wired communication network or a wireless communication network, with other mobile agents 100, supervisory systems, or external computing devices by utilizing transmission media 220. In some embodiments the mobile agents 100 operate without a wireless communication network, relying solely on passive visual sensing of one or more neighboring mobile agent. The non-transitory computer readable medium 218 may store a computation model 218a and control logic 218b as discussed herein. The computation model 218a is utilized for processing visual data and may be of insect retina to motor neuron dynamics. The control logic 218b is utilized to direct the movement ofthe mobile agent 100, and may be derived from a theoretical multi-agent behavior (convergence boundedness) analysis, that causes the at least one processor 208 to analyze, at instants of time within the time period, said visual information according to said computational model 218a to generate actuator commands. The actuator 204 may include propulsion motors, rotors, wheels, controlsurfaces, thrusters, or other mechanical actuation devices configured to modify the motion state of the mobile agent 100.
[0200] The processor 208 executes the computation model 218a and control logic 218b to generate actuator commands. Motion blur corresponds to streak patterns generated by relative motion of neighboring mobile agents 100 across the image plane. As used herein, a minimum motion blur magnitude refers to a threshold value below which detected blur responses are treated as insignificant or noise. In some embodiments, the control logic 218b is configured to, at instants of time within the time period, cause the at least one processor 208 to detect, motion blur within the visual information caused by the at least one neighboring mobile agent 100, calculate a bearing angle (the angular direction between the mobile agent 100 and a neighboring mobile agent 100 measured relative to the mobile agent body frame) at which a specific feature in the motion blur occurs within the visual information and generate actuator commands based on the bearing angle. The specific feature in the motion blur is one or more of: a peak motion blur magnitude, a peak motion blur minimum magnitude, a blur inflection point, or blur flat points. A peak motion blur magnitude corresponds to a location in the visual field exhibiting maximal intensity change due to relative motion. A peak motion blur minimum magnitude corresponds to a minimum allowable magnitude of such a detected peak used to distinguish valid motion cues from insignificant blur responses. A blur inflection point is a point in a motion-blur (intensity profile) where the rate of change of blur magnitude transitions between increasing and decreasing behavior. A blur flat point corresponds to a region of the motion-blur intensity profile where the spatial gradient approaches zero, indicating a locally constant motion blur magnitude. The processor 208 computes optic flow from the visual information to determine relative motion between the mobile agent 100 and surrounding agents. In some embodiments, once the neighboring mobile agent 100 producing the maximum optic flow magnitude has been identified, the control logic 218b applies a pure pursuit guidance rule in which the mobile agent 100 steers toward the bearing angle of the selected neighboring agent 100. In this manner, the optic-flow-based detection of the neighboring agent 100 is converted directly into a motion control command that adjusts the orientation of the mobile agent 100 toward the selected target, i.e., the selected neighboring agent 100.
[0201] The computational model 218a may implement an approximation of insect retina-to-motor neuron dynamics using WFI operators. The control logic 218b may include one or morecontrol algorithms, navigation routines, sensor interpretation logic, and permanent memory used by the mobile agent 100 during operation. The sensory system 202, actuator 204, processor 208, communication interface 212, and memory 218 may be interconnected via a communication pathway or internal data bus 228 that enables exchange of data among the components of the mobile agent 100. The actuator 204 executes the actuator commands generated by the processor 208 to achieve and maintain a local formation or coordinated motion with at least one neighboring mobile agent 100.
[0202] FIG. 3 illustrates an example visual data structure and transformation pipeline 300 performed by components of the mobile agent 100 for processing visual information associated with the mobile agent 100 and generating a structured representation usable for control, navigation, or decision-making to achieve and maintain a local formation or coordinated motion with at least one neighboring mobile agent 100, through actuator commands. The control and navigation may be accomplished by the processor 208 generating and providing control signals to the actuator elements 204 to cause the actuator elements 204 to guide the mobile agent 100 in a predetermined manner coordinated with the neighboring agent 100. Coordinated motion emerges from the distributed visual interactions between mobile agents 100.
[0203] FIG. 3 illustrates an exemplary visual-data processing architecture executed by the processor 208 of the mobile agent 100. In this architecture, raw visual input 305, such as image intensity I and optic flow CD, is first received from the sensory system 202 and subjected to preprocessing 310 to perform filtering, normalization, or noise reduction. The processed visual information is then analyzed in a feature extraction stage 320 to determine motion-related quantities such as angular motion Q and orientation 0. These extracted features are used to compute derived state variables 330, including quantities such as position x, velocity v, and relative distance p. The derived states are then used to determine control parameters 340, such as control gains K and control inputs u, which influence subsequent motion control of the mobile agent 100. In some embodiments, outputs from the stages of the pipeline are aggregated into a structured representation 350 that stores the variables generated during processing.
[0204] In some embodiments, visual information is received as raw visual input 305 received by the sensory system 202, which may include image data, optical flow measurements, depth maps, or other visual sensing outputs represented by variables such as I (pixel intensity as afunction of position and time) and < D (optic flow with components of horizontal and vertical image velocity). The raw visual input may be obtained from the sensory system 202 of the mobile agent 100 by the processor 208 of the mobile agent 100. As discussed above, the sensory system 202 may include cameras, optical flow sensors, event cameras, or other visual sensing devices such as radar, lidar or the like.
[0205] The raw visual input 305 may be provided to a preprocessing stage 310 implemented within the control logic 218b being executed by the processor 208. The preprocessing stage 310 may perform operations such as filtering, normalization, noise reduction, temporal smoothing, or other signal conditioning processes that prepare the raw visual data for further analysis.
[0206] The preprocessed visual information may then be provided to a feature extraction stage 320 implemented by the control logic 218b executed by the processor 208. The feature extraction stage 320 may determine one or more visual features or descriptors derived from the visual input. In some embodiments, the extracted features may include quantities represented by variables such as 0 (an orientation angle) and Q (change in 0 over time), although other feature representations may also be used. These features may represent motion cues, directional information, visual flow characteristics, or other properties derived from the visual observations.
[0207] The extracted features may be used to determine one or more derived state variables 330 associated with the mobile agent 100 or its environment including the one or more neighboring mobile agent 100. In some embodiments, the derived states may include quantities such as x, v, and p, which may correspond to state variables such as position, velocity, relative position, distance, or other state representations. The processor 208 determines optic flow from the visual information to determine relative motion between the mobile agent 100 and neighboring mobile agents 100, comprising an image-velocity vector field representing motion of visual features across the image plane. The derived states 330 may be computed using estimation methods, transformation functions, or other computational processes that infer state information from the extracted visual features.
[0208] In some embodiments, the derived states 330 may be used by the processor 208 to determine control parameters 340. The control parameters 340 may include quantities such as K and u, which may correspond to control gains, control inputs, command signals, or other parameters used to influence the behavior or motion of the mobile agent 100. The controlparameters may be determined according to one or more control laws, feedback mechanisms, or control algorithms that utilize the derived state information 330.
[0209] Outputs produced by one or more stages of the pipeline 300 may be aggregated into a data structure 350. In some embodiments, the data structure 350 may include a collection of variables such as (t, I, CD, O, 0, x, v, p, K, u), although other combinations of variables may also be used. The variable t may represent a time value associated with the captured data. The data structure 350 may store or represent visual inputs, extracted features, derived state variables, control parameters, or other associated information in a unified representation.
[0210] The data structure 350 may be used by the processor 208, as well as additional processing modules, control systems, navigation systems, or decision-making systems associated with the mobile agent 100. In some embodiments, the data structure 350 may be stored, transmitted with the communication interfaces 212, or used to update control actions of the mobile agent 100.
[0211] Although FIG. 3 illustrates one example arrangement of processing stages, the particular order, number, and configuration of stages may vary in different implementations. In some embodiments, certain stages may be combined, omitted, or executed in parallel. Additional processing steps may also be included without departing from the scope of the disclosure.
[0212] The term "optic flow" refers to the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and the scene. The geometry of optic flow definition is illustrated in FIG.4, showing the relationship between an observer (also referred to herein as an observing agent) 400 (e.g., an uninformed mobile agent 110) and a target 406 (e.g., an informed mobile agent 105 or an uninformed mobile agent 110) in a spherical coordinate system. The term "wide-field integration" refers to a mathematical operation that extracts motion cues from optic flow by computing spatial inner products with predetermined basis functions. The term "small-target motion detector" or " STMD" refers to a neural mechanism, or engineered analog thereof, that responds preferentially to small moving targets in the visual field.
[0213] The present disclosure encompasses multiple embodiments, including: (1) a first feedback control system based on wide-field integration of optic flow for motion coordination in multi-agent groups; (2) a second feedback control system based on STMD neurons for target tracking and group motion with minimal connectivity; (3) robustness analysis and gain inmodulation methods for maintaining performance under sensor noise and configuration uncertainties; (4) optic flow enrichment methods using head or camera motion to enhance observability in low-signal environments; (5) swarm guidance methods using high-speed "streaker" agents (i.e., a subset of informed mobile agents 105) that are intentionally fast moving relative to uninformed agents 110 to ensure that the streaker agent is selected as a neighboring agent to be followed by uninformed agents 110 for implicit information transfer; (6) spacecraft and satellite applications; and (7) aerial vehicle implementations of the foregoing systems. The following describes non-limiting examples of the presently disclosed inventive concepts.Definition of Optic flow modeling
[0214] Disclosed herein is a parallel approach to the derivation of optic flow formulations for solitary environments like corridors and tunnels, and instead derive optic flow for use in a multi-agent context by considering the optic flow experienced by an individual mobile agent 100 (observing agent 100) surrounded by other agents 100 in motion. The discrete optic flow Qjfor a finite number of objects j located at distances rj > 0 from the observing agent 100 utilized isQj= -ω × Qj- [v - ⟨v, Qj⟩Qj], (2.1.1)where <o G IR3is the angular velocity of the observer, Qy is the unit vector in the direction of the target object, and v G IR3is the linear velocity of the observer. This model can be represented in a continuous, body-frame-relative spherical coordinate system with the following:
[0215] The matrices A and B are nonlinear functions of observer-relative azimuth y and elevation ft angle to the target shown in FIG.4, and are given by
[0216] See FIG. 4 for the geometry of the optic flow definition. To robustly handle large distances, it is convenient to replace the inverse distance function — with a nearness function, H, which is dependent upon the viewing angles y and ft between the observer and the objects in its surrounding environment, as well as its own states qwhere r > 0 remains the distance between the mobile agent and the surrounding objects.
[0217] After deriving the general 6-DOF vector expressions, they are specialized to a planar environment having three configuration variables q = (x,y, 0) and optic flow can be expressed as(xsiny — ycosy).
[0218] For an environment where multiple mobile agents 100 are present, the optic flow is determined by the state (qb, q&) of the mobile agent 100 as well as the states of all the other mobile agents 100 in its neighborhood (qt,qt), where t = [1,2,...,n], and n is the total number of neighboring agents 100.
[0219] In this multi-agent environment, the components of the optic flow equation are functions of the dynamic states of mobile agents 100 in the neighborhood (referred to as targets within the present disclosure), relative to a particular bio-inspired agent 100 (also referred to as bio-agent or mobile agent within the present disclosure ). The rotational component of optic flow includes the angular velocity wtof the target, with its radial part discarded, scaled by the nearness function, which reflects the greater contribution of the targets near the bio-agent 100.
[0220] For horizontal planar motion in a 3-D (planar) environment, the mobile agent states are again restricted to q = (x,y, 0) and q = (xb,yb, 0). Under such conditions, the normal component of optic flow Qp becomes zero and the tangential component is expressed as(2 1 2)+ Y> qi> qt t - i>)siny - (yt- y cosy)}.The optic flow for each mobile agent 100 thus becomes a piecewise continuous 27r-periodic function of the viewing angle yt, with the number of discontinuities being proportional to thetarget agents 100 in the environment. We assume no collision between mobile agents 100, thus r(y, qb, qt) 0 and / i(y, qb, qt) G (0, oo) can be generally expressed as(2.1.3)where rbtis a vector connecting the two agents.Optic-flow based feedback control in multi-agent flight
[0221] In this section, an idealized case of the mobile agent 100a (referred to herein as a bioinspired agent or bio -agent) with arbitrary initial conditions (y, 0 ) within a group of mobile agents lOOb-e whose velocity has already converged, as shown in FIG. 5 (Geometric parameters of the mobile agent 100a in idealized group of mobile agents lOOb-e), is considered, and a feedback rule and its properties is developed.
[0222] Using Eqn. 2.1.2 and Eqn. 3.3.3 from the previous section, the mobile agent 100a implements an established feedback control law based on wide-field integration (WFI) of optic flow. The mobile agent 100a can be constructed in a similar manner as the mobile agent 100 discussed above. Specifically, in this embodiment, as discussed below, the control logic 218b can be implemented as follows to generate actuator commands based on the WFI outputs. The lobula plate tangential cells of the visuomotor system of insects process the complex patterns of optic flow during motion to extract relevant information about its surroundings for navigation. The computational model 218a executed by the processor 208 implements this wide-field integration. The output of these neural cells can be modeled as a spatial inner product in a function space L2(S2 / S2)aswhere Fj(y) is the set of weight functions (weighting responses) that represents the cells' sensitivity to optic flow patterns illustrated in FIGS. 6A and 6B. These WFI outputs map the retinal motion patterns into sensory outputs comprised of perturbations from the desired optic flow and have been measured experimentally and also designed using control-design tools.
[0223] In this study, we use WFI outputs as feedback to drive the mobile agent 100a to detect neighboring mobile agents lOOb-e and maintain the swarm formation during flight, asmodeled in FIG. 7. In particular, FIG. 7 shows a block diagram of a closed feedback system 700 having blocks 702, 704, 706 and 708 in series. For an orientation imbalance correction, sensitivity functions are chosen F±= cos(y) and F2= cos(2y) and the feedback control input provided to block 702, scaled by some gains Ktin block 708, is thus given byu = K f ) + K2z2(x), (2.1.5)WhereZi < Q,cos(y) >, z2=< Q, cos(2y) >. (2.1.6)
[0224] An optic flow calculation is accomplished in block 704, and the results of the optic flow calculation are supplied to a wide-field integration calculation accomplished in block 706 to create a WFI output. Applied to a differential-drive ground vehicle set at a constant forward speed vb, the system 700 uses Eqn. 2.1.5 (input) as the closed-loop feedback for orientation control. The dynamic model, linearized along the reference trajectory xref = (vb, 0,0) to decouple the forward speed dynamics from the (y, 0) states, given in the statespace form is expressed asStability analysis of the dynamic model
[0225] In this section, we analyze the stability of the closed loop system 700 linearized about the nominal (centered) reference condition using a n definition specific to this configuration.
[0226] The nearness function, n(y, qb, qt), for 2D planar motion in a multi-agent formation shown in FIG. 5 can also be written as a function of the viewing angle, y, and the states of the mobile agent 100a and the mobile agents lOOb-e surrounding the mobile agent 100a as ysin(y+0b)0 < y + 0b< H)
[0227] In contrast to the general case in Eqn. 3.3.3, Eqn. 2.1.8 is singular at (0b+ y) G {0,7r}. The non-holonomic condition in Eqn. 2.1.7 restricts the singularity from occurring unless the mobile agent lOOb-d lies directly ahead or behind the mobile agent 100a and the mobile agents lOOb-d have parallel velocity vectors, thus for considering small perturbations from the reference trajectory, Eqn. 2.1.8 remains valid. For this system 700, we choose without loss of generality a coordinate frame with its x-axis along the group's motion, with the mobile agent lOOa's states xb= (xb,y,9) and traveling along the reference trajectory xb=(ub,0,0). Given a finite number of mobile agentslOOb-e, for example, the non-linear optic flow equation (Eqn. 2.1.2) and closed loop control feedback input u (Eqn. 2.1.5), the WFI output (Eqn. 2.1.6) may be written as followswhere n = total number of mobile agents lOOb-d, i.e., target agents, and
[0228] For the formation shown in FIG. 5, let the relative velocity along the x-axis be (%t— %!,) = xr, and the relative velocity along the y-axis be (yt— yb) = yr, and define a and b as the half width of the formation such thaty3 - yi y4 ~y2, x3- x±%4- x2a = — — > b = — - — = — - —.2 2 2 2These assumptions greatly simplify the WFI equation (Eqn. 2.1.6).
[0229] Assuming a slight perturbation in orientation, A0, the feedback equation is linearized along the equilibrium reference trajectory (yb, 0b, b> ®b) = (0,0, v^O). For an established formation of target agents, (yt, 0t, xt, 6t~) = (yt, 0, vt, 0). This yieldswhere Ci_6are constant functions of yt.
[0230] Similarly, for a slight perturbation in position, Ay, the feedback equation z2linearized along the equilibrium reference trajectory and an established mobile, i.e., target, agent 100b-e formation yieldsUsing these values to analyze response of the dynamic model in Eqn. 2.1.7 due to perturbations. The system 700 can be expressed asand WFI-output of the system 700 (Eqn. 2.6.1) is expressed asWherefor Ci_9being constant functions of yt, which has characteristic equations2- p12s - vbpi= 0,and eigenvaluesFor a choice of positive relative velocities along with the given assumptions, it is clear that the choice of gains must be K1< 0 and K2< 0.Observability Analysis of WFI Output
[0231] For the system 700 described above for finite number of mobile agents 100 in a swarm, observability analysis will determine how well the states can be measured from the WFI output. This is particularly important for this model since the states are used in the feedback control input, and the optic flow output is used to estimate the states. Using the dynamic model in the linear optic flow equation (Eqn. 2.1.9) the observability matrix rPn P12 1,,2= QVbP11jhasdeterminant Vup^.
[0232] The system 700 is observable forp1; L0 since vb> 0 for all t > 0, and p1= 0 only when xr= 0. That is, only when vb= vt, the determinant of the observability matrixbecomes zero, the matrix ceases to be full rank and the system 700 is no longer observable. This unobservability result is a consequence of the optic flow equation (Eqn. 2.1.2) and shows that optic flow depends upon the relative motion relative to the bio-agent's surroundings, which encodes an intuition that can have significant consequences on the resulting behavior. Simulation
[0233] Utilizing a simplified (consensus) swarm environment of a finite number of mobile agents 100 flying in 3D using a planar version of Cucker-Smale model, the simulation environment uses the interaction method proposed by using the distance of all neighboring mobile agents 100 and their velocities to update its own velocity. This model assumes that each mobile agent 100 updates its velocity by integrating it with a weighted average of its relative velocity with all the other mobile agents 100 in the swarm. That is, for time fc £ N and agent b and swarm agents (i.e., mobile agents) n,where the weights wbtare a function of the distances between the mobile agentswbt= *1 CH xb- xt||2),and Tj is a non-increasing function depicting the loss of influence of swarm agents (i.e., mobile agents) with increase in relative distance and is given byfor some K, a > 0 and? > 0.
[0234] This formulation eventually gives rise to directional ordering and flocking behavior. This model is suitable for highly polarized flocks that attain high mean direction of motion within their group and is less applicable to swarming insects that show lower polarization levels.
[0235] Thereafter, insects have been experimentally shown to use optic flow for flight navigation, which we compute ideally in the simulation environment. The initial optic flow for the mobile agent 100a in a circle of neighboring mobile agents lOOb-e as a function of the viewing angle ytproduces a sine wave pattern FIG. 8A. A change in relative velocity xrfromthat state produces a change in the amplitude of the output (See FIG. 8B), while a change in the position, Ax or Ay, causes an asymmetry in the curve (See FIG. 8C), and phase shift is produced by orientation adjustments A0b(See FIG. 8D).
[0236] The computed optic flow is used along with the set of basis functions to compute WFI outputs for a single mobile agent, i.e., the mobile agent 100b, the mobile agent 100c, the mobile agent lOOd, and the mobile agent lOOe within this swarm environment. Initially, we have reduced the dynamics to planar motion with an x-y axis lateral motion and yaw-axis rotation. The WFI outputs for the bio-inspired agent (i.e., the mobile agent 100a) is used as control feedback for regulation.Simulation Results
[0237] FIGS. 8A - 8C are graphs illustrating piecewise optic flow pattern (a) with some relative velocity at initial conditions, (b) for relative velocity change, (c) for lateral position change, and (d) for orientation change of the mobile agent 100a. FIGS. 9A-9C illustrate the results of this simulation. For this simulation study, we illustrate using a group of 50 mobile agents lOOb-n (arbitrary n). The simulation environment implements the planar Cucker-Smale (C-S) model for planar motion, with cr = 200 and ft = 0.45 chosen arbitrarily, and K a function of cr and ft that satisfies the stability constraints described above. The C-S model adjusts the linear velocity of agents lOOa-n to achieve convergence over time, and the WFI outputs are used as feedback to regulate the velocity of the bio-inspired agent 100a based on its computation of the idealized optic flow around it. The simulations indicate convergence of the bio-inspired agent 100a is achieved along with the other mobile agents lOOb-n in the group. The output (FIG. 9A) shows how the bio-agent 100a changes its initial velocity, as a response to the other agents lOOb-n, to match their velocity and it maintains that velocity as t -> oo, participating in the swarm motion. The distances between the bio-agent 100a and the other agents lOOb-n (FIG. 9B) are bounded and the bio-agent 100a follows the swarm (FIG.9C). This convergence behavior was maintained when adding an additional bio-inspired feedback agent to the simulation, and that agent also participated similarly.
[0238] FIGS. 10A-10C are graphs illustrating orientation regulation of bio-inspired agent 100a, from an initial offset of 0 = 45 °, using idealized optic flow computation for WFI feedback in a group of 50 mobile agents lOOb-n as seen in change in orientation offset over time (a), bounded distance between agents (b) and motion trajectories (c).
[0239] The set of simulations discussed above used WFI outputs for feedback navigation in a modified C-S model for emergent group behavior. Next, a simulation environment is created to replicate a fully formed swarm behavior of 50 agents lOOb-n with velocity convergence and bounded distances between agents lOOb-n already established at time t = 0. The bioinspired agent 100a with the feedback control model developed previously (Eqn. 2.1.7) is introduced to this environment and its performance is analyzed. The bio-agent 100a has an initial orientation misalignment of 45° and is positioned at the center of the swarm. The results shown in FIG. 13 demonstrate that the bio-inspired agent 100a rectifies its heading until it aligns itself with the direction of motion of the swarm agents lOOb-n, although it is clear from FIG. 10A that the offset correction takes longer than the previous simulation environment and there is no visible centering response. The inter-agent distances are bounded (see FIG. 10B), and the bio-agent 100a shows no further perturbations to its heading once it reaches a stable equilibrium state. This behavior was not affected by varying the number of swarm agents lOOb-n in the bio-agent's local neighborhood.Experimental ImplementationHardware
[0240] The theoretical model and simulations indicate that with the appropriate selection of basis functions, bio-inspired optic flow feedback may be useful for orientation and speed regulation in a swarm. For experimental implementation, shown in FIG. 11 is a Robotarium testbed, a remote-access multi-robot research facility in Georgia Tech, was used. The testbed arena is 3m x 2m in dimension and hosts differential drive microbots called Gritsbot that are 0.11 m wide, 0.10 m long, and 0.7 m tall with sensors that accurately track position and velocity data and wirelessly communicate that with a global host.
[0241] The Gritsbots are not equipped with a vision sensor, and thus for implementation of the dynamic model for orientation regulation, a simulated sensor (visual) is used in which the mathematical equations of optic flow derived in previous sections are used to calculate idealized optic flow around the bio-inspired agent 100a. The onboard Atmega 328 chip computes this idealized optic flow along with wide-field integration outputs based on the states of all the mobile agents 100. Even though this reduces effects due to sensor noise, the performance of the experiment will be hindered due to actuator limitations, communication delays, accuracy and resolution of the onboard sensors and achievable closed-loop bandwidth. Some physical parameters of the robots in the Robotarium arena include amaximum linear velocity of 50 cm / s, maximum rotational speed of 9 rad / s and a collision avoidance algorithm execution when any inter-robot euclidean distance is less than 12 cm. Experimental Results
[0242] Two cases have been explored in this setup. In the first case, the bio-inspired agent 100a is set to have a lower forward speed as that of the other agents lOOb-n in the swarm, whereas for the second case the forward speeds are set to be equal.
[0243] FIGS. 12A-C illustrate the first case, wherein the bio-inspired agent 100a is set to have a lowerforward speed as that of the other agents lOOb-n in the swarm, i.e. |vb| |vt|. It can be clearly seen from FIG. 12C that the bio-agent 100a corrects its initial orientation offset to reach the equilibrium condition of 0 = 0 within a short amount of time, and then maintains its motion along the direction of the swarm. The planar optic flow generated due to the swarm agents lOOb-n for one experiment run of the first case is shown in FIG. 13, which shows that the planar optic flow experienced by the bio-inspired agent 100a is initially large due to the orientation offset resulting in a high relative velocity of the bio-agent 100a with respect to the swarm. This causes the WFI outputs, and in turn the angular velocity response of the bioinspired agent 100a, to be high in magnitude, leading to the corrective feedback response of the bio-agent 100a predicted by the theoretical model. The magnitude of the angular velocity eventually decreases as the bio-agent 100a reduces the orientation offset with respect to the swarm direction of motion, resulting in a decline in the optic flow generated by the other agents lOOb-n. As the bio-agent 100a reaches stability, the control feedback input reduces to maintain the attained desired trajectory. Since the bio-agent 100a never matches the swarm speed in this case, a component of optic flow is always generated since the relative motion with the surrounding agents lOOb-n does not go to zero, resulting in slight perturbations in its orientation 0 value for the rest of the experiment.
[0244] FIGS. 14A-C Illustrate experimental results (the mean orientation, mean angular velocity and mean trajectory of the bio-agent 100a for 20 trials of the second case) of the second case wherein the forward speeds of the bio-inspired agent 100a and the swarm agents lOOb-n are set to be equal, i.e. |vb| = |vt|. For both embodiments, the bio-inspired agent 100a robot calculates the idealized optic flow using the non-linear mathematical model derived in the optic flow equation (Eqn. 2.1.2), which is then used to compute WFI outputs for navigation. The following is the mean and variance of 20 trials with these set of initial conditions: bio-agent orientation offset 0O= 45° at ( o<yo) = (—1-35,0) and the rest of theswarm agents (robots) surrounding the bio-agent 100a at zero initial orientation offset with a fixed forward speed of 8 cm / s along the x-axis to imitate a swarm that is already formed, analogous to the simulation case previously discussed in FIGS. 10A-C. The bio-agent 100a has a fixed forward speed of 7 cm / s and 8 cm / s respectively for the two cases. Gains K1= —0.5 and K2= —0.2 provided acceptable overshoot and rise time.Experimental Results
[0245] In this section, the results of the Robotarium experimentation are presented and discussed. The mean orientation, mean angular velocity and mean trajectory of the bio-agent 100a for the 20 trials of the first case where |vb| |vt| is shown in FIGS. 12A-12C. It can be seen from FIG. 12C the bio-agent 100a corrects its initial orientation offset to reach the equilibrium condition of 0 = 0 within a short amount of time, and then maintains its motion along the direction of the swarm. The planar optic flow generated due to the swarm agents lOOb-n for one experiment run of the first case is shown in, which shows that the planar optic flow experienced by the bio-inspired agent 100a is initially large due to the orientation offset resulting in a high relative velocity of the bio-agent 100a with respect to the swarm. This causes the WFI outputs, and in turn the angular velocity response of the bio-inspired agent 100a, to be high in magnitude, leading to the corrective feedback response of the bio-agent 100a predicted by the theoretical model. The magnitude of the angular velocity eventually decreases as the bio-agent 100a reduces the orientation offset with respect to the swarm direction of motion, resulting in a decline in the optic flow generated by the other agents lOOb-n. As the bio-agent 100a reaches stability, the control feedback input reduces to maintain the attained desired trajectory. Since the bio-agent 100a never matches the swarm speed in this case, a component of optic flow is always generated since the relative motion with the surrounding agents lOOb-n does not go to zero, resulting in slight perturbations in its orientation 0 value for the rest of the experiment.
[0246] FIGS. 14A-14C illustrate the mean orientation, mean angular velocity and mean trajectory of the bio-agent 100a for 20 trials of the second case where |vb| = |vt|. The outputs of this case are very similar to that of the previous case where the velocities are not equal. The bio-agent 100a apparently seems to show corrective behavior to align its orientation with that of the surrounding swarm agents, and attain equilibrium conditions. The mean trajectory plot in FIG. 14C shows the bio-agent 100a attaining and maintaining equilibrium conditions for every trial run. FIG. 15 shows the planar optic flow generated dueto the swarm agents lOOb-n for one experiment run of the second case. The optic flow input, as well as the WFI output, goes to zero as equilibrium state is reached, since there is no relative motion caused by the swarm agents lOOb-n in the bio-inspired agent's surroundings. Discussion
[0247] The simulation results suggest that it could be possible to participate in an established swarm behavior by integrating optic flow feedback into the control of multi-agent systems. This result begins to address the ability of current bio-inspired individual feedback models in a swarm context by showing that engineered swarm systems can impose swarm motion on bio-inspired agents 100a. This test complements previous observations on animals where a small minority of fast-flying "streaker" bees guide the entire swarm when moving to a new site, navigating fish and birds. With the presence of a minority of "informed" agents, the results indicate that the bio-inspired agents 100a are able to achieve group flight in the direction of motion of the informed swarm agents. This transfer of motion from a majority of informed agents to minority of mobile agents implementing individual feedback rules complements previous studies highlighting information transfer from a minority of informed individuals. Besides understanding the context and performance of these agents by considering larger numbers of bio-inspired agents 100a into the simulation environments, considering the effects of the weight functions governing the bio-inspired feedback control may reveal some additional degrees of freedom to adjust swarm behaviors.
[0248] The orientation time-history FIG. 12 indicates that the robot initially overshoots, then reaches stability over the course of the experiment. Apart from the K1and K2gains not being optimal, this overshoot can emerge from limitations in the physical hardware parameters such as communication and processing delays, tracking system precision, actuator constraints and the overall directional change of the swarm configuration overtime, as visible from FIG.12C.
[0249] Once the bio-agent 100a reaches equilibrium conditions at 300 time steps shown in FIGS. 14A-C for the experiment with |vb| = |vt|, the bio-agent 100a does not show any corrective feedback response, as can be seen in the angular velocity time history in FIG. 14B. This behavior supports the linearized mathematical model stability analysis which showed that the linearized system becomes unobservable once the relative velocities xr= vt— vband yr= yt— ybbecome zero. The planar optic flow time history for each of the mobile agents lOOb-n for one trial run in FIG. 15 shows the optic flow input reducing to zero at 300time steps, resulting in the loss of any WFI outputs and feedback control. As a result, no feedback regulation by the bio-agent 100a is observed from then on. This indicates a necessity of the presence of relative motion for an agent to navigate in a swarm. This also suggests traditional formation flight may be difficult to achieve in a group using optic flow, as that will render the system to be unobservable at the reference point, compared to the less polarized swarm motion behavior we observe in insects that continuously stimulates optic flow.
[0250] For comparison, two other controllers are implemented in our simulation environment: an idealized proportional controller and a vision-based parallel-formation controller developed in Moshtagh et. Al IEEE Transactions on Robotics 25 (2009), no. 4, 851-860. The proportional controller assumes perfect information is shared over communication links and uses the difference between the reference trajectory and the system output as the error signal, and its control input is proportional to that error signal u = Kp* {q*(t, qt) — q(t)}, where q* denotes a desired trajectory and q denotes the states of the bioagent. The proportional controller gives an ideal output trajectory (FIG. 11.14) with regulation of both (y, 0). For the controller proposed in Moshtagh, all the mobile agents implement the same controller with the collision avoidance term proposed in that paper, see FIGS. 16A and 16B. The feedback input in Moshtagh is defined for each mobile agent i as
[0251] where tj is thee to collision for some gain K < 0. atis the collision avoidance term based on a choice of potential function that is a function of inter-agent distances. The mobile agents in this system "choose" the direction of parallel motion based on the initial conditions and control gains. Without the collision avoidance term, the mobile agents showed segmentation for certain initial conditions. This controller did not show convergence to the consensus travel direction when only one agent used the controller, without the collision avoidance term, in an established swarm motion of other agents (Fig. 16B). In comparison, the visuomotor controller studied in this disclosure shows orientation regulation to achieve swarm tracking and requires established group motion in its neighborhood. When compared to the idealized controller, the bio-agent controller follows a similar trajectory for orientation regulation but does not show explicit centering behavior.Discussion of assumptions and implications
[0252] Insects are thought to compute optic flow in a different manner than typical engineered systems, relying on mechanisms like elementary motion detectors rather than approaches like Lucas-Kanade. Insect visuomotor sensors do not necessarily compute optic flow with the fidelity of established iterative computational optic flow sensing algorithms such as Lucas-Kanade, suggesting that a simpler estimation of optic flow could better resemble its biological counterpart.
[0253] This study isolates the effects of the feedback approach, rather than these differences in sensing, by using idealized optic flow calculated during the experiments instead of choosing a specific implementation of optic flow sensing and computation. To allow readily reproducible results, experimental verification of the system was conducted in an openaccess research facility using idealized optic flow on micro-bots that also lacked optical sensors. The resulting conclusions, particularly the observability requirements, are thus general in nature and are not isolated to a particular optic flow sensor implementation.
[0254] While the study began with the formulation of a nonlinear optic flow equation and nearness function for general 3-D dynamic environments and autonomous feedback control system for planar motion, the stable equilibrium point was predicted by a linearized model. Formally, a linearized result is sufficient to establish only a locally stable equilibrium, and a nonlinear analysis would be needed to examine the region of stability of the feedback control system or establish global stability. Nonetheless, the results were qualitatively insensitive to the variation in swarm densities we reproduced in simulation and experimental studies.
[0255] The chosen nearness function for the linearized model is singular for agents lOOb-n directly ahead or behind the bio-agent 100a with their velocity vectors parallel to each other. This function is specific to the established swarm formation in FIG. 5, and remains valid for small perturbations from the reference equilibrium condition. A choice of n as a function of cosine or only the states x and y of the mobile agents lOOb-n can be used to define n at the singularity condition for a more general case of swarm formation.
[0256] The experimental swarm agents' angular velocities were set to zero by assuming straight trajectories without orientation perturbations to validate a simplified model of the system. The experimental implementation experienced small uncommanded trajectory variations in some of the target agents lOOb-n, rendering a greater variance in the mean trajectories. These deviations were included in the experimental runs to imitate a changingtrajectory rogue agent behavior commonly found in individuals within a swarm. The fact that the bio-agent 100a can reorient itself about the equilibrium point even after slight inconsistencies in the target agents' swarm pattern is an indication of the robustness of the model, suggesting some robustness and stability of the proposed feedback controller, and at the same time allows others to readily implement the approach with imperfect target tracking.
[0257] Future experimental improvements may help identify some features related to sensing and the role of collision avoidance. Theoretically no collision between the mobile agents lOOb-n was assumed to ensure the optic flow measurements do not reach infinity due to loss of inter-agent spacing, and the experimental work included this low level collision avoidance assumption for practical reasons. Improvements to include optic flow sensors and multiple bio-inspired agents 100a within a group of other target agents lOOb-n in higher dimensional spaces (e.g., 6-DOF flight) will be useful in isolating the role that a particular choice of optic flow computation technique can play in this environment and in evaluating the feedback control approach for swarm formation and collision avoidance. However, these future results may be specific to the choice of sensor and optic flow implementation. The current results and their implications for optic flow techniques apply independent of sensor implementation.Summary and conclusions
[0258] Within the present disclosure, contemporary bio-inspired visual feedback modeling was examined with respect to agent behavior in groups. A control theoretic model of dynamic visuomotor feedback in a swarm context is proposed based on wide-field integration of optic flow feedback. Analysis of that model based on a set of basis functions and feedback gains predicts stability about a reference trajectory such that the bio-agent 100a rejects perturbations from a relative reference position based on the optic flow pattern induced by the swarm agents lOOb-n around it. Simulated and experimental results demonstrate orientation alignment of the bio-agent 100a with the direction of motion of the surrounding agents lOOb-n. This result indicates that visuomotor processing and feedback may have an influence on the formation of group motion in insects. Comparable results from simulations using 50 agents and experiments using 5 agents suggest robustness of the theoretical model to the number of mobile agents 100b-n participating in group motion. Comparison with similarmodels from prior art shows differences in outputs suggesting a role of WFI in following swarm motion.
[0259] The presence of relative motion proved to be a crucial factor in system observability and performance, suggesting group behavior in insects could be characterized by greater variance in direction and speed of individuals within the swarm as compared to formation flight in flocks of birds or fishes with high polarization but little relative motion, since the former offers greater optic flow cues for navigation. Thus, this insect visuomotor feedback study suggests that while an insect-inspired optic flow feedback approach may participate in more traditional flocking and swarm behaviors, the implications of the underlying visual sensing mechanism may indicate that there are fundamental differences in the behavior of swarms generated by optic-flow and feature-tracking feedback approaches.Robustness and the Gain Modulation HypothesisIntroduction
[0260] Real world applications of multi-agent autonomous systems often deal with unmodeled and uncertain circumstances such as changing lighting conditions, background noise, mechanical failure, sensing delays, unmodeled dynamics, disturbances and obstacles in the environment. Performance of autonomous feedback control frameworks can also suffer from linearization of nonlinear characteristics of the system, unmodeled higher-order dynamics of the plant or sensors and deviation of the plant from the idealized conditions assumed during modeling. Robust control analysis and design techniques take into account the specific uncertainties affecting the system's behavior and include tools to establish stability margins for the system in presence of uncertainties.
[0261] This disclosure presents bio-inspired multi-agent visuomotor feedback control approach (managing visual interaction between agents) developed previously and applies the available robustness analysis tools such as Hmcontrol optimization to improve the system's performance in the presence of uncertainties.
[0262] The main contributions of this disclosure include (a) robustness analysis of an existing bio-inspired visuomotor feedback approach based on the wide-field neurons that integrate optic flow in insect visuomotor system during multi-agent flight, (b) experimentation of the bio-inspired feedback model on ground robots equipped with camera sensors for optic flow measurements, (c) implementation of the robustness conditions in experiments to analyze performance of the robust system and (d) the introduction of a novel working hypothesis thatinsects ensure robust swarm flight performance in part by sensory gain modulation, and practical implementation of said behavior to a swarm of autonomous or semi-autonomous agents.Visual Feedback Control System with LPTC Model
[0263] Shown in FIG. 17A is another closed feedback system 1700 constructed in accordance with the present disclosure. The closed feedback system 1700 is similar in construction and function as the closed feedback system 700 discussed above, but has been modified to represent idealized (noise-free) sensors in FIG. 17A. Similar blocks have been labeled in FIG.17A for purposes of clarity. Wide-field integration (WFI) of optic flow measurements are used as feedback to navigate the controlled agent 100a among neighboring agents lOOb-n and maintain the swarm formation during flight, as modeled in FIG. 17A. The simplified model assumes horizontal planar motion where the neighboring agents lOOb-n already have an established formation while the controlled agent (referred to as the bio-agent 100a for the rest of the section) regulates its trajectory for some initial conditions, as shown in FIG. 17B. Boundary walls are not considered in designing the feedback control algorithm, neither are the presence of obstacles in the environment. White, texture-less boundary wall in the experimental testbed (FIG. 19C) ensures minimum optic flow signal from the background, thus any background optic flow signal is attributed to sensor noise. Previous experimental results computed idealized optic flow signals from neighboring agents' states without the use of any visual sensors. In this section, this idealized experimental result is used as the nominal case due to the lack of sensor noise.
[0264] The optic flow detected by the bio-agent 100a due to its own motion and the motion of one neighboring agent lOOb-n can be written as a function of both of their stateswhere bio-agent states are qb= xb,yb,0 and qb= (xb,yb, 0), neighboring agent states are qt= (xt, yt, 6t~) and qt= (xt, yt, 0t), y is the bearing angle and the nearness function, n, can be generally expressed aswhere rbtis a vector connecting the two agents. Assuming no collision between agents at any given time, rbt= 0 and p G (0, oo).
[0265] The optic flow calculation in an insect is not ideal, and is instead computed by elementary motion detectors (EMDs) which may be modeled as Hassenstein-Reichart cross correlation detectors. In the present disclosure it will be sufficient to consider the frequency domain sensitivity. To consider this, a first-order transfer function is augmented to the optic flow sensor block. The first-order transfer function for elementary motion detector sensitivity to target velocity is chosen as(2.2.3)
[0266] Wide-field integration of optic flow is performed by the lobula plate tangential cells (LPTC) of the visuomotor system of insects to extract relevant information about its surroundings from the input optic flow pattern and can be modeled as a spatial inner product in a function space L2(S2, IR2) aswhere£(y) is the set of weight functions that represents the cells' sensitivity to various optic flow patterns.
[0267] The insect dynamic model considered is a lateral-directional Drosophila. The second-order lateral insect flight dynamics is determined byir -9.69 0 9.81 0 0 8.240P -9720.0 -177.0 0 -2.7 0 12300.0 00 1 0 0 0 0 o rdir -167.0 462.0 0 -71.0 0 0 -28100.0 0 0 0 0 1 0 0 0 y-1 0 0 0 vb0 0(2.2.5)
[0268] where the state vector is composed of sideslip v, roll rate p, bank angle 0, heading rate r, heading angle 0, and lateral position y is written in terms of previous state values and the control inputs: wingstroke amplitude asymmetry and stroke angle asymmetry fid. Theinclusion of two additional states i and y, result in a six state, two inputs and six outputs system for each mobile agent 100.
[0269] A differential drive ground robot model is often a more applicable platform for experimental testing of the sensor noise characteristics in controlled laboratory environments. When the system in FIG. 7 is implemented on ground vehicles that follow the non-holonomic constraint, the resulting linearized system can be expressed asand WFI-output of the system (Eqn. 2.1.6) is expressed aswhere, vb= constant forward speed of bio-agent,for Ci-9 being constant functions of the bearing angle yt. This model includes two states, one input and two outputs for the ground robot, and is chosen to be consistent with idealized sensor experiments earlier described.Uncertainty modeling framework
[0270] A more refined picture of the feedback control system 700 in FIG. 7 for real-world applications includes signal and system uncertainties. Sources of uncertainty include parametric perturbations, faulty sensors and actuators, computational delay in feedback, noise and image distortion in the visual sensor, error in optic flow calculation, external disturbances such as winds, and unmodeled system nonlinearities. This work includes the effect of noise in the measurement system, and multi-agent configuration uncertainties. The feedback control system 1700 for multi-agents with uncertainties can be generally represented by the block diagram in FIG. 17A, in which blocks 1704, 1706 and 1708 providing a scaling function Wo, uncertainty A and noise n have been added. This framework is used for the robust analysis in this study to include signal and system uncertainties. FIG.17B illustrates a generalized model 1720 of system uncertainties.Modeling Measurement Noise
[0271] Using a camera sensor to provide visual inputs to the bio-agent 100a for optic flow calculations is a prime source of measurement uncertainty affecting the system output. The optic flow measurement noise coming into the system from the camera sensor was estimated experimentally. The bio-agent 100a is placed in the experimental testbed with the target agents lOOb-n around it in formation and optic flow is computed for the stationary agents lOOa-n. The lack of relative motion means idealized optic flow values are zero at all points, and any non-zero optic flow output in this condition was considered sensor noise.The system with only measurement signal uncertainty corresponds to FIG. 17A with Wo= 0.The noise-to-error transfer function for the system with measurement uncertainty is given bywhere K is some arbitrary negative-definite gain matrix, P represents the mobile agents dynamics, [OF] represents the optic flow sensor dynamics and [W Z] represents the wide-field integration block. Noise rejection on this system may be analyzed by application of Hmloop shaping. To account for these uncertainties and measurement noise, the control logic 218b can be implemented using said Hmloop shaping to calculate the actuator commands, in which the loop transfer function is given byL(s) = FP[0F][ F / ]. (2.2.8)The noise-to-error transfer function changes to include the system uncertainty loop when Wo0. However, for robustness analysis, the model uncertainties are not considered when analysing the robustness to external disturbances.Modeling System Uncertainty
[0272] The desired swarm formation in FIG. 5 used for the feedback law derivation is an approximation and changes during the experiment (FIG. 18), leading to a system uncertainty. In this framework, velocity is fixed and the bio-agent 100a directly regulates only its orientation. Consequently, lateral position offset is the primary mechanism causing deviation from ideal multi-agent formation and was considered as the primary unmodeled uncertainty parameter representing configuration changes. Since this multi-agent configuration variation only affects the output of the optic flow measurements and not the mobile agent dynamics, it is modeled as an unstructured output multiplicative model uncertainty, shown in FIG. 17Aassuming zero sensor noise ns. Experimental data suggests the configuration variation is a function of the relative speed between the bio-agent 100a and the other agents lOOb-n, and the size of the uncertainty is a function of speed. Robust control theory has developed several tools to map this framework to the generalized MA-structure system uncertainty model in FIG. 17B.
[0273] To transform the system 1700 in FIG. 17A to the generalized model of system uncertainties 1720 in FIG. 17B, the loop at the pointsand yAis broken and the transfer function matrix as 'seen' by the uncertain block A is calculated:rule
[0014] )where GT= P[0F], KT= K'fPV’F / ]. The transfer matrix GTKT(I + Gr / Cr)-1can be denoted as the output complementary sensitivity function Toand thus, from FIG. 17B.M = —W0T0.
[0274] The weights in Woprovide an idea of the 'size' of model uncertainty A as a function of frequency. To accurately model the weights Woto define the model uncertainties affecting the system, the size of the change in formation during the experiments is analysed. The nearness function, n, defined in Eqn. 2.2.2, can also be written as
[0275] Eqn. 2.2.9 suggests that uncertainty Adfin distance dtwill translate to an uncertainty in the nearness function n, which in turn introduces uncertainty in the optic flow measurements based on the optic flow equation in Eqn. 2.2.1. A conservative assumption is that Hi is directly proportional to Qi, the optic flow generated by the target agents. Thus, weight transfer functions for uncertainty in the optic flow sensor output is chosen based on the experimental data for uncertainty introduced by Adi.
[0276] For the generalized unstructured output uncertainty model in FIG. 17B, theTTa, norm of A is assumed to be unknown but stable and normalized to be ||1. The diagonal transfer function weight matrix Wois designed to reflect the likely spectral content of the uncertainty in the system. The weights are chosen to be stable minimum phase transfer functions. From the Small-Gain theorem for output multiplicative uncertainty, the uncertain system in FIG. 17B is stable if and only ifSince A is defined to be normalized and || A1,which gives the robust stability condition(2.2.10)
[0277] Assuming weights to be unity (Wo= I), the || Tonorm dictates the robustness of the feedback system. In other words, the smaller thenorm of To, the higher the uncertainty (and Wo) can be before the system reaches instability.
[0278] In the case of sensor noise ns= 0, Eqn. 2.2.10 still holds. The output complementary sensitivity function Tochanges to include the noise term. However, for robustness analysis, external disturbances are not considered when analyzing the robustness to model uncertainty.Experimental Implementation (Embodiment)
[0279] For real-time experimental implementation on ground robots, five ground-based mobile agents 100 running onboard decentralized feedback (FIGS. 19A-19C) are used, starting with the formation shown in Fig.l9C. The mobile agents 100, i.e., target agents, are equippedwith markers (FIG. 19B) 2010 and the experiment arena has low visual texture (FIG. 19C ) to ensure only the swarm motions are generating the measured optic flow. The ground-based mobile agents 100, in this embodiment, operate on Raspberry Pi 3b running Raspbian OS. Python, C++, and ROS Kinetic are used to implement the onboard feedback system processes operating on the processor 208, including onboard image acquisition and processing, feedback control calculation and implementation. The mobile agents 100 in this embodiment have a maximum linear velocity of 22 cm / s and maximum rotational velocity of 2.84 rad / s. No explicit collision avoidance algorithm is implemented on the ground-based mobile agents 100. The experimental arena shown in FIG. 19C consists of a 8x 12 ft enclosure with white boundary walls that minimize unwanted background optic flow signals.
[0280] See FIG. 19A, the ground-based mobile agent 100 (bio-agent) in this embodiment is fitted with a single panoramic 360° camera 1905. The camera 1905 used is Picam360 with a vertical FOV of 220° and a horizontal FOV of 360° (FIG. 20A). It captures 640 x480 pixel images at 20 fps. The raw, circular image is then distortion corrected into a panoramic image using image processing (FIG. 20B). The Picam360 is mounted face down to ensure the target agents occupied a large FOV in the de-warped images. A single horizontal line from the input image contains all the target agents and is used to perform onboard optic flow calculation (Fig.20C). Optic flow calculation from the camera image input is performed using the Gunnar Farneback algorithm using the opencv apps ROS package. The Gunnar Farneback algorithm calculates dense optic flow at fixed points in the visual input based on pixel intensity changes between successive image frames.
[0281] The ground-based mobile agent agents 100 are decentralized and do not require communication among each other. A remote desktop running Linux-based Ubuntu 16.4 was used to initializes all the mobile agents 100 at the start of the experiment. Each experimental trial consisted of 15 seconds.
[0282] Two cases have been explored in this setup. In the first case, the feedback control gains are chosen arbitrarily from the set of acceptable gains calculated previously, K1= —0.3 and K2= —0.002. Different initial angular offsets, A0O, are applied to observe robustness to varying initial conditions. For the second case, the gains are reduced as previously indicated in robustness analysis of the nominal system in order to mitigate the effect of noise and model uncertainty on the sensor outputs, K = —0.01 and K2= —0.001. Only one particular initial condition from the first case is selected for comparison. For both cases, the forward speed ofthe ground-based mobile agents 100 are set to 9 cm / s and 11 cm / s for the bio-agent 100a and the target agents lOOb-n (respectively) as relative motion ensures observability. The surrounding target agents lOOb-n are placed in the formation shown in FIG. 5 with no angular bias and a fixed forward speed to duplicate motion in an established formation.
[0283] This study presents the multi-agent feedback system's robustness analysis output based on experimental data, along with 20 trials each of the two cases with this set of initial conditions: bio-agent 0O= 45° in ( o<yo) =with a constant forward speed of 9 cm / s. The mean output of 5 trials for each of the different initial angular offsets of the first case are also presented.Results and Discussion
[0284] In this section, the results of robustness analysis and experimental implementation of the bio-inspired multi-agent visual feedback system are presented and discussed. The experimental results for the nominal case are based on the findings in Robotarium testbed with ideal optic flow calculation, and robustness measures are determined by incorporating noise and configuration uncertainty into the experimental implementation.
[0285] Optic flow measurement noise nsduring flight experiments can generally be defined as zero mean, E{ns} = 0, with known covariance E{nsiiJ} = Rns. To model camera sensor noise in this study, the probability density function of the optic flow measurement noise signal calculated by the stationary bio-agent 100a at one image point containing a stationary target agent was approximately Gaussian with zero mean and a peak (over all trials) standard deviation of 0.4.Robust analysis
[0286] The ground robot experiments represent a simplified implementation of the multiagent system and will be analyzed first. The singular value plots of the loop transfer function (Eqn. 2.2.8) of the nominal ground vehicle dynamic system (FIG. 21A) indicate the linearized plant model is insensitive to frequency changes, which follows from the simplifications of a kinematic model applying linearized optic flow equations with no sensor dynamics.
[0287] To improve the resolution of this analysis, the kinematic model of the ground vehicle is then replaced by the lateral insect flight dynamics Eqn. (2.2.5) and the elementary motion detector filter in Eqn. (2.2.3) is included on the sensor outputs. The model including insect and sensor dynamics is then analyzed for robustness.
[0288] The singular values of the loop transfer function in Eqn. 2.2.8 for gain K = — I is shown in FIG. 21B. For high frequency, this system satisfies the noise rejection property of fL) « 1. In this case, 50% noise rejection ( -6dB gain) can be attained for frequencies over 180 rad / s. The static gains in the gain matrix K can be lowered to attain better noise rejection at lower frequencies based on design specifications. Choosing the diagonal elements of gain matrix K = —0.3 and —0.01 improved the noise rejection at the desired operating frequency of 30 rad / s. FIG. 21B also indicates good tracking performance cannot be achieved with this system since the criteria o(L) » 1 cannot be met for low frequencies. The extremely low values of the minimum singular value aminare due to the wide-field integration process that amplifies input signals from one particular direction and nullifies inputs from its orthogonal direction; this is consistent with experimentation.
[0289] To model the weighting transfer functions in Wofor output multiplicative model uncertainty, experimental data from the nominal case is considered. The weighting matrix Wois a m x m diagonal matrix where m is the number of target agents in the surroundings. Data suggests higher relative speed between the bio-agent and the target agents leads to a greater value. Wois chosen (conservatively) to have the same weights on the uncertainty for all s, which are found to reach a maximum of |A / = 50% at low frequencies based on the experimental trials. Weights are then chosen to attain 100% at 30 rad / s and peak at 200% for high frequencies. The resulting weight transfer function is given by (FIG. 22A).2s+30i0= (2.2.11)s+60
[0290] FIG. 22B shows the singular value plot of the output complementary sensitivity function To. For this system with static gains K = —I,II To!!«,= Ill,which implies the system is stable for uncertainty0.009,which is significantly less than the uncertainty we expect from our experimental results. Again, reducing the static gains in the feedback law increases the bound on uncertainty, and in turn the bounds on / z.
[0291] Based on the weight function in Eqn. 2.2.11, and reducing (static) gains, the robustness condition in Eqn. 2.2.10 is met, as shown in FIG. 22C.
[0292] The mean orientation and mean angular velocity of the bio-agent, along with the mean trajectories of all the mobile agents for 20 trials of the first case are analyzed. As in the results of the nominal case, the bio-agent regulates its initial angular offset of 0O= 45° to turn towards the motion of the established swarm and maintains its direction once equilibrium angular value 0* = 0° is reached, FIG. 23A indicates a high variance in the 0 value near the equilibrium point, and some overshoot by the bio-agent. Since the bio-agent does not match the forward speed of the target agents, a component of optic flow is always generated by the target agents. The angular velocity 0 plot in FIG. 23B results show the control input to the bio-agent does not go to zero once equilibrium is reached, which is likely due to the presence of optic flow measurement noise even after 0 = 0° is reached. FIG. 23C shows the mean trajectory of the bio-agent and swarm agents.
[0293] FIG. 24 shows the mean orientation plot from FIG. 23A and that of the nominal case for the same initial conditions of 0O= 45°. Both the plots show similar convergence behaviors. The nominal case uses idealized optic flow calculation from known state values of the target agents and thus has a lower variance than the feedback system in FIGS. 23A and 23B, especially during the initial overshoot at about t = 550. The variance in the results from the nominal case are likely caused only by the model uncertainties discussed previously.
[0294] Experimental results containing sensor noise vs nominal (Robotarium) results illustrate the effects of sensor noise. Regarding the mean orientation plot from and that of the nominal case for the same initial conditions of 0O= 45°, both the plots show similar convergence behaviors. The nominal case uses idealized optic flow calculation from known state values of the target agents and thus has a lower variance than the feedback system previously discussed, especially during the initial overshoot at about t = 550. The variance in the results from the nominal case are likely caused only by the model uncertainties discussed earlier, since a perfect sensor performance is assumed. The use of camera sensor for optic flow calculation introduces additional uncertainties in the feedback control input for the experiments in this disclosure.
[0295] The mean orientation and mean trajectories of 5 trials for each of the different initial angular offsets of the first case are shown in FIGS. 25A and 25B. The initial angular offsets are chosen as 0O= [180°, 135°, 90°, 45°, -45°, -90°, -135°]. For 0O= 180°, the bio-agentmoved in either direction arbitrarily to regulate its orientation towards the swarm motion; only one directional trajectory is plotted. The bio-agent shows robustness to varying initial conditions by correcting its trajectory towards the established swarm motion, although the orientation plots suggest the feedback system became more unstable with increasing values of |0O|. This could be due to the fact that by the time the bio-agent 'detects' and regulates its orientation for higher 10O| values, the target agents have moved further away from the bioagent resulting in larger model uncertainty in the system. The mobile agents do not show lateral position regulation, as indicated by the nominal case output.
[0296] The mean orientation and mean angular velocity of the bio-agent, along with the mean trajectories of all the mobile agents for 20 trials of the second case, shown in FIGS. 26A-26C, demonstrated that similarly to the first case, the bio-agent shows orientation regulation to reach equilibrium trajectory of 0* = 0°. It is clear from FIG. 26C that the variance in the trajectory of the bio-agent has significantly decreased with the lower gains, as expected from the robustness analysis. The orientation plot in FIG. 26A suggests increased damping leading to no overshoot as in the first case, which also ensures lower variance and smoother convergence to equilibrium.Discussion
[0297] Sensor measurement noise, varying lighting conditions and luminosity, lens distortion introducing image warping, accuracy of the optic flow algorithm, input disturbances at the plant actuators, varying battery levels of the mobile agents for different experimental runs, unmodeled higher order dynamics of the camera sensor and deviation of the mobile agents from the idealized agent formation are all sources of uncertainties that affect the performance of the multi-agent visual feedback system. Since the multi-agent system is decentralized without any need for inter-agent communication, input disturbances due to communication delay, bandwidth limitations or loss of communication with the remote terminal can be assumed to be nonexistent. The robustness analysis for the bioinspired visual feedback system in FIG. 7, modified to have insect lateral flight dynamics in the nominal plant model, identifies the major sources of signal and system uncertainty as measurement sensor noise and varying multi-agent configuration.
[0298] Attenuation of measurement noise signals at the plant output requires a(T0) to be small. Robust stability to unstructured output multiplicative uncertainty also requires o'CT’o) to be small. Analysis of the closed-loop feedback system indicates that the stabilityrequirements can be met by tuning the gains to be lower. Since other performance requirements are not considered, designing dynamic controllers using techniques such Hmcontroller design or / z-synthesis is not necessary. Gain modulation is also fairly common in insect neural networks and this modulation may be part of the reason insects achieve robust flight maneuvers with a fairly simple collection of neurons. The singular value plots in FIGS.22A-22B and FIGS. 23A-23B show significantly low minimum singular values ( a ) for the nominal system, introduced by the characteristic of the wide-field integration of optic flow in the feedback loop. The output of this spatial inner product function is ideally zero when the optic flow due to the target agents on either side of the bio-agent is balanced, resulting in the output of the [WF / ] block being zero for certain input directions and amplified for others. This characteristic of the wide-field integration process determines the shape of the a plot for the system.
[0299] Experimental results for the nominal plant indicate that the addition of camera sensor for optic flow measurements introduces signal noise into the system input. Input sensor noise may account for the oscillatory behavior of the orientation plots in FIGS.26Aand 26B as it reaches the equilibrium trajectory. At the low frequency that the system is operating in, the higher-order camera sensor dynamics does not seem to have much affect on the measurements. By design, the bio-agent should maintain its direction of motion (and feedback should be zero) when the optic flow on either side is balanced, and this is satisfied even when the bio-agent is initially moving in the opposite direction of the established swarm, as in the case of 0O= 180°. The bio-agent continues to maintain the initial angular offset of 0O= 180° for some time until some slight deviation from that trajectory triggers the feedback control to regulate the orientation towards the motion of the established swarm.
[0300] Experimental results from the second case (FIGS. 26A and 26B) indicates attenuation of noise signals and stability to model uncertainty for the desired operating frequency can be achieved by lowering the gains. The trade-off is tracking performance. Reduced tracking performance may be acceptable in many biological contexts, may contribute to the richness of a signal, and flight records for tracking tasks often include oscillations that indicate lower gains than typical engineering approach. As can be seen in FIG. 26A, the bio-agent does not quite reach the equilibrium point of 0* = 0° and as a result has greater deviation from the reference trajectory (FIG. 26C). Lateral position regulation is not performed for this multiagent system. This results in greater model uncertainty being introduced in the systemcompared to the first case. The bio-agent apparently reaches equilibrium much more smoothly and without any overshoot, and has tighter bounds on the variance, thus validating the theoretical robustness results from earlier.
[0301] The results from this section suggests feedback gain modulation aids in robust performance in presence of sensor and system uncertainties. Although a majority of biological measurements have identified gain modulation in the sensor ratherthan a controller module, tight integration in biological systems suggests this could achieve the same goal. For example, given our system model 700 in FIG. 7, feedback gain modulation in the feedback control input has the same effect as gain modulation in the visual sensors. Based on the Hmrobustness analysis results of this section, we introduce the hypothesis that insects perform sensory gain modulation to ensure robust performance during a wide variety of flight conditions and uncertain environments. This hypothesis leads to testable and measurable artifacts, including a parallel reduction in tracking performance.
[0302] In multi-agent experiments, one must distinguish the effect of surrounding agents from environmental responses such as the solitary behaviors observed in previous work, such as speed regulation and tunnel-axis alignment. This study has used theory and experiments to isolate the multi-agent behaviors, however, many experiments will contain contributions from both behaviors. One example is experiments in a converging-diverging tunnel: in this case, solitary study has documented tunnel axis alignment. In the environment-free multiagent case, this study documents swarm alignment. Because a realistic optic flow field contains contributions from both environment and multi-agent behaviors (e.g., both tunnel axis alignment and swarm alignment), these behaviors may generally compete in a manner dependent on the specific geometry, feedback gain, and visual texture of targets.
[0303] Two major sources of uncertainty are selected for robustness analysis in this section: sensor noise and varying agent configuration. There are many other sources of uncertainties that can affect the performance of a multi-agent feedback system including communication delay, faulty sensors and actuators, computational delay in feedback, visual sensor image distortion, error in optic flow calculation and unmodeled system nonlinearities.
[0304] Since the mobile agents in the multi-agent system in some embodiments only use their own states and visual-feedback for navigation, no inter-agent communication links were required for pose information sharing. Uncertainties due to loss of communication links, network topology or limited bandwidth are thus not necessary for this result.
[0305] The individual plant models are not considered as major sources of plant uncertainty in this analysis. The Ground-based mobile agents used in the experiments are equipped with Raspberry Pi 3b computers running at 1.4GHz processor speed that executes the feedback control algorithm on-board the mobile agents. The mobile agents are also equipped with alert systems to notify users when the battery runs low. The ground vehicles are also checked for potential faulty parts at the start of experimentation. All these features ensure that the effects of uncertainties due to computational delay, battery drainage and faulty actuators or sensors are considerably lower than the major sources of uncertainties identified.
[0306] The Picam360 used as the camera sensor in the bio-agent captures circular images which are then distortion-corrected by a simple lens distortion algorithm to extract a panoramic image. The distortion mapping was not able to remove all of the distortion. However, the mapping errors were concentrated at the bounds of the image and no measurable effect was identified on the horizontal range used in the experiment.
[0307] While optic flow algorithms do not always map to idealized optic flow, the Gunnar Farneback algorithm used for optic flow calculation has lower average error compared to the more popular and widely-used Lucas-Kanade algorithm. These effects were not able to be isolated from the identified noise in the visual sensor.
[0308] Modeling of the noise signal and the uncertainty weights were based on the actual data collected during experiment and the bounds on these uncertainties may vary for different scenarios such as changes in background, lighting and other testbed physical properties.
[0309] A linearized model of the system is considered for robustness analysis. This approach means some of the higher-order system dynamics may have remained unmodeled and therefore the results do not necessarily give any indication to the system's scalability for increased number of target agents or how different background environments affect the system performance. For improved understanding of this multi-agent feedback system's region of stability, non-linear stability analysis techniques may be applicable.
[0310] A robotic experiment cannot replicate all aspects of the biology and some differences in biological analysis and robotic experiment are inevitable. In this case, the robust biological analysis was an insect flight dynamic model that includes sensor dynamics, while the robotic experiments were implemented on ground robots in a controlled laboratory setting. The underlying sensing and feedback rules were held consistent for both the theoretical flightsystem and the ground robot experiments, to provide a method for validating the robustness results. Nonetheless, the ability to consistently prescribe initial and environmental conditions with a known (fixed) gain allows interrogating the robotic experiment in a way that is not generally possible in biology, and the consistent results between the two approaches are helpful.Summary and Conclusions
[0311] In this section, a bio-inspired multi-agent visual feedback control system is implemented on a different set of mobile agents equipped with camera sensors for optic flow measurement. Nominal performance for different gain levels is compared with that of the computed idealized optic flow results. The outcomes indicate similar performance to the nominal system with greater uncertainty entering the feedback control input. The nominal system is also stable over a wide range of initial angular offsets for the bio-agent. The inclusion of camera sensor for optic flow measurements introduces uncertainties. A robustness analysis is performed that includes consideration of sensor (measurement) noise and output model uncertainty to predict stability over a range of norm bounded uncertainties. Results indicate static gains picked from the set of acceptable gains can be tuned to reduce the effect of measurement signal noise and increase the allowable model uncertainty range. Study of the small target-sensitive neurons in predatory dragonflies suggests insects could use gain modulation during flight. The proposed gain modulation hypothesis and the results from this study are applicable to understanding how insects can robustly achieve flight behaviors during group motion.Mitigating Position Blindness via Optic Flow EnhancementIntroduction
[0312] Previous sections of this disclosure expanded the in-flight individual insect feedback rule to a multi-agent system using the baseline multi-agent wide-field integration of optic flow. An observability criteria identified from the theoretical stability analysis stated the importance of relative motion between the mobile agents (vb= vt). The wide-field integration of optic flow based multi-agent system developed in this disclosure assumes the neighboring agents generate strong optic flow signals necessary to extract relevant information for motion coordination. Environments with low relative motion can lead to weak optic flow signals and complicate the application of optic flow based navigation algorithms. Flight in sparse environments also degrades optic flow signals. Examples of these challengingcontexts include low texture environments like salt flats and swarm regulation with sparsely distributed agents, yet experimental evidence shows that insects retain performance in these conditions where engineered systems' performance degrades.
[0313] This section considers an optic flow enrichment study and addresses the question: is the established head rotation model, used as a tool to enrich optic flow signal in low signal environments, robust enough to be implemented on higher-fidelity lateral-directional Drosophila flight dynamics to enable position regulation?BackgroundProblem formulation
[0314] The head motion study assumes a flying insect (the 'viewer', denoted by subscript b) tracking another insect (the 'target', denoted by subscript t) flying at a constant positive velocity vton the line y = 0 in the (x,y) inertial plane as illustrated in FIG. 27 and uses the kinematic model of ground vehicles with non-holonomic constraints (Eqn. 2.1.7) as the plant model. The velocity of the viewer is assumed to be always parallel to its body x-axis (sideslip is negligible). The controllersand UQ are feedback laws designed as functions of the optic flow experienced by the viewer. The viewer is then expected to regulate its speed, orientation and position with respect to the target using only the controllersand UQ which are functions of the optic flow input. The resulting closed-loop feedback system is shown in FIG.28.
[0315] FIG. 28 illustrates a block diagram of an example closed-loop control architecture 2800 configured to regulate motion of a viewer agent relative to a target using optic-flow-derived feedback. The system receives target states 2805 from a target agent, e.g., the mobile agent 100b and viewer states 2810 from a viewer agent, e.g., the mobile agent 100a, which are combined at a summation node 2815a to generate a relative state signal representing relative motion between the mobile agent 100a and the mobile agent 100b. The relative state signal is processed by an optic flow computation module 704 configured to determine an optic flow quantity Q representing the apparent motion of the target across the viewer's visual field.
[0316] The optic flow signal Qis provided to signal-processing modules HSs2820a and HSC, 2820b which generate respective directional components of optic flow represented by Qsin yand Qcos y. These components represent projections of the optic flow signal alongorthogonal directions associated with the bearing angle / between the mobile agent 100a (viewer) and the mobile agent 100b (target).
[0317] The directional components are applied to gain elements 708 K1, K2, K3, and K4, which generate weighted feedback signals that are combined through summation nodes 2815b and 2815c to produce control commands. The summed signals generate control inputs u^and uv, corresponding respectively to orientation control and velocity control applied to the viewer agent. These control inputs drive the viewer dynamics, resulting in updated viewer states 2810 that are fed back to the summation node 2815a, thereby closing the feedback loop. Through this feedback structure, optic-flow measurements are converted into actuator commands that regulate the viewer’s motion (mobile agent 100a) relative to the target (mobile agent 100b).Feedback controllers
[0318] The optic flow enrichment formulation includes the outputs of two neural cells in the insect lobula. These outputs correspond to the target located at an angle y(t) at a time instant t, and is given bya = QftSin / ft, (2.3.1)
[0319] Using these two signals, the output feedback laws are chosen asue= k3a4+ k4b2., (2.3.4)
[0320] The feedback equations are linearized at the reference trajectory Xe= [veyeAxe0e]r= [vt00.50]ryielding the linear feedback laws with time varying coefficientsa-u ( = a (t) (b- vt) + ay (t) (yb-yt) + (2.3.5)a&(t)( xb- A%0) + a-e(t 0b ~ 0t)>bu (t) = bv(t) (vb- Vt) + by (t) (yb- yt) + (2.3.6)
[0321] The full expressions of these time-varying coefficients can be referred to from Mehdi Yadipour, Md Arif Billah, and Imraan A Faruque, Optic flow enrichment via drosophila head and retina motions to support inflight position regulation, Journal of Theoretical Biology (2023), 111416 (hereinafter Yadipour et al.). The optic flow equation used for deriving these coefficients assume constant insect head rotation rate a> with an arbitrary radius r. The stabilizing, time-varying output feedback gains are calculated using a linear quadratic regulator (LQ. R) at each time step At = 0.01s. The two feedback laws will be used with the insect lateral-directional dynamics for speed and position regulation in the next subsection.
[0322] This body of work establishes the intimate connection of the underlying physics of insect flight dynamics with sensing, specifically in reducing sensing requirements. This context limits the generality of conclusions that may be reached from a study considering only sensory structures.
[0323] To consider how insect flight dynamics affects this result, we replaced the nonholonomic dynamics in with an insect lateral flight dynamics model adapted from augmented with first-order forward speed dynamics. The linear model was x(t) = Ax(t) + Bu(t) withwhere sideslip v, roll rate p, and bank angle evolve as a function of inputs(differential stroke amplitude) and pd(differential stroke plane), and ubis reference forward flight speed with dynamics ub= 14501^. Stroke plane adjustments are a primary kinematic modulation associated with yaw motion studies, thus the previous controller was retained with lateral outputs mapped to pdand speed regulation to Up.
[0324] The simulation includes the insect lateral directional flight dynamics and parameters sideslip velocity (v), roll rate (p), bank angle (0), yaw rate (r), heading angle (ip), lateral position (y), differential wing-stroke amplitude (CD), differential stroke-plane angle ( ), forward flight speed (vb) from Table II.1 are seen in FIGS. 29A-29H. The result shows that the stability in 0b, yb, and ubis not destroyed, and out-of-plane motions are at least two ordersof magnitude smaller than in-plane. Lateral position remains convergent, while relative range is bounded by forward speed damping, which is partially explained by optic flow's stronger sensitivity to deviations in lateral position versus longitudinal.Table II.1: Simulation parameters ( i indicates initial time)Discussion
[0325] This study presents a potential mechanism by which cruising insects could make use of head rotations to improve optic-flow based distance regulation. FIGS. 29A-29H show the insect regulates its forward speed to the desired 1.2m / s and corrects its orientation deviation from 20° to 0° within 5 seconds. Position regulation is also seen as ybgoes to 0 from its initial offset. The results from FIGS.29A-29H suggest that the optic flow enrichment via head / retina motion formulation can be translated from horizontal planar motion in kinematic models to insect dynamics. In simulation, the bio-inspired agent regulates its forward speed toward a desired reference value while correcting orientation error and maintaining bounded lateral position using optic-flow-based feedback with motion enrichment.Discussion of assumptions and implications
[0326] The optic flow model used in this work is posed for nonholonomic planar motion and was retained for the more generalized insect flight dynamics motion study, including the computation of the target's angular position Yij • This limitation is mitigated by the generalized motion simulation remaining largely planar, with out of plane motions remaining less than two orders of magnitude smaller than in-plane motions: peak roll angle is less than 3 x 10-3rad / s, and sideslip is less than 5 x 10-5, while heading 0breaches angles of 20°. Similarly, the planar simulation considers both speed and orientation regulation, while the insect flight dynamics simulation incorporates lateral directional dynamics, studying an orientationregulation pathway without speed and position regulation. Similarly, the longitudinal dynamics are simplified to a single affine differential equation in speed. Because an outcome of planar head motion speed enrichment is to strengthen position regulation, incorporating longitudinal dynamics could add significant perspective.Summary and conclusions
[0327] This section considers a flying insect and a moving target. Head rotation is introduced to modify the optic flow measurements, enriching the observability of body motion states. This section then uses a formulation in which a flying insect regulates its body states relative to another flying agent using only optic flow outputs, which is a configuration that has previously been shown to provide a challenging optic flow field. An established engineering model of the insect visuomotor system as an output feedback controller is implemented on a lateral-directional Drosophila flight dynamics. This model has previously been applied as a model of the insect optomotor response and includes position regulation challenges in poor optic flow fields. The stability of the planar motion using kinematic model is retained in insect lateral-directional flight dynamics simulations. These results demonstrate how head motions can be used to enrich an optic flow field that previously provided challenging results to the feedback regulation, and provide a tool for understanding the role of the head motions measured in flying insects. The results show that optic flow produced by head motions could be useful for an insect to regulate its speed and position relative to another insect in environments with weak optic flow signals.HIGH IMPACT RESPONSE TO INDIVIDUALSIntroduction
[0328] Insects robustly achieve the coordination needed for collective group flight, largely by leveraging a small collection of neurons in their visuomotor system. Tethered insect studies have identified individual neurons in the visual sensory pathway and their respective roles in environmental navigation. Despite the progress in identifying the insect visuomotor neural pathways, the resulting flight control functions of these neural processes are not always known, particularly when stimulated by large numbers of neighboring agents.
[0329] Some insects demonstrate high visual acuity during interaction with prey, predator or conspecific neighbors. This interaction requires visual detection of targets in unstructured environments. Targets often subtend only a small region of the visual field relative to sensor resolution. The small-target motion detector (STMD) neurons found in the lobula complex ofcertain insects such as dragonflies, blowflies and hoverflies are highly sensitive to small, dark, fast-moving objects. Models predicting the behavior of STMDs suggest that these neural models can be used for target detection and pursuit in artificial vision-based robotic applications. Although recent progress in multi-agent modeling has shown how a parallel wide-field visual processing pathway can participate in swarm motion, there is not a clear understanding for how the STMDs behave in a multi-agent environment. This study investigates whether the responses of STMD neurons can help insects participate in swarm motion through theory, simulation, and experiment. The main contributions of this paper include (a) a concise retina-to-motor neuron dynamic model of STMD neuronal feedback, which goes beyond the STMD neuronal models as simply sensory mechanisms, (b) analysis and stability proof of state boundedness over time, necessary for reliable robotic implementations, (c) simulation and analysis of information flow showing low levels of connectivity, indicating low attentional requirements relative to contemporary approaches, and (d) experimental work using external recording of onboard implementation in robotic hardware, used to provide an uncertainty bound by showing repetitive behavior.Methods and Approach
[0330] In the following paragraphs, the modeling, analysis and validation methods for the multi-agent feedback control system is described. A simplified model of the STMD neuron function is proposed, a multi-agent switched feedback control system using the STMD function is developed, stability analysis of the designed control system is provided followed by implementation in simulation and on ground robots.Small-Target Motion Detector (STMD) Neuronal Model
[0331] The magnitude of optic flow signal depends upon the size, relative velocity and the distance between the observer (e.g., mobile agent 100a) and the target (e.g., mobile agent 100b), along with the observer's own motion. Literature shows the STMD neurons found in insects are also highly responsive to a target's size, velocity, contrast, relative distance and direction of motion. The STMD neurons also respond to a single target at any particular time. From the cascaded model in Wiedermann and O' Carroll, it is evident that the neural pathways for both optic flow and STMDs can be correlated or even be assumed to be cascaded one after the other in an EMD-ESTMD formation. In this formulation, the STMD neurons process the optic flow patterns to extract relevant information about the surroundings. Based on the properties of STMD neurons, this study proposes the output of STMD neurons bemathematically modeled as mathematical maxima (max) and argument of maxima (arg max) operators on the optic flow signal from its preceding neural pathway. Accordingly, the computation model 218a can be implemented to determine this maximum optic flow magnitude. The STMD-model thus identifies the maximum magnitude of optic flow signal and its corresponding angular position from the entire visual field of view
[0332] where Q is the optic flow signal calculated by the computation model 218a defined in a continuous, body-frame-relative spherical coordinate system as developed, and y and ft are the respective azimuth and elevation angles as shown in FIG. 4. This proposed STMD-model captures the STMD neurons' dependency on target's size, velocity, relative distance and direction of motion, along with the single target selection property of the STMD neurons.
[0333] For initial analysis of the implications of Eqn.3.3.1 in a multi-agent system, the STMD-model is implemented in a group of homogeneous agents (in simulation and ground robots experiments) constrained to horizontal planar motion and the STMD neurons are assumed to be perfect sensors with no temporal or frequency dependence, resulting in zero time delay and signal amplification over a wide range of frequencies. Boundary walls or obstacles are not considered.
[0334] For horizontal planar motion, y spans between [— 7r,7r], (3 = 0 and the planar optic flow equation for multi-agent environment can be expressed as in Billah and Faruque,3 3 2))cosy)},where (q*, 4b) and (qt, qt) are the states of the viewer and the target agents respectively and the nearness function n is the inverse of the magnitude of the vector rbtconnecting the viewer and the target agents, expressed asBio-inspired Multi-Agent Feedback Control System
[0335] The STMD-model detects and outputs the selected target agent, in terms of the maximum optic flow magnitude (ystmd,i) and its corresponding angular position (ystmd,2)- Afeedback law to close the loop is then necessary to govern how the STMD-output is used to regulate the mobile agent's motion. Dragonflies use STMD neurons during flight to pursue and encounter tiny prey and conspecifics and a feedback guidance law from insect pursuerevader models often termed as 'pure pursuit' strategy in literature was chosen to define the higher-order neural feedback following the STMD-output. As discussed below, the control logic 218b evaluates the target bearing and generates actuator commands that are transmitted to the actuator 204 using these pursuit guidance laws, while enforcing an average dwell time constraint to prevent system instability. An exemplary feedback rule executed by the control logic 218b is given byui ~ ^[ystmd,2]j—arg max(Qi) (3.3.4)Y,is the feedback law for ithsubsystem, i = 1,2,..., N, and K > 0 is an arbitrary scalar feedback gain. This proportional feedback rule structure aims to maintain a constant bearing angle between the target and the pursuer.
[0336] FIG. 30 illustrates a Small-target motion reactive (STMR) 3000, a bio-inspired multiagent closed-loop feedback model based on STMD neurons as sensory mechanism. The selection of one 'target' agent is based on the STMD-model output 3005, which results in a continuous-time switching system where the selection of target by a single agent switch between different neighboring agents overtime depending on the highest magnitude of optic flow (determined by the optic flow calculation 704) generated by sensing the neighboring agents by the sensor system 202. The feedback law 3010 converts the target-detection outputs of the STMD module into control inputs that drive the controlled agent dynamics 3015, thereby adjusting the motion of the mobile agent 100a, for example, in response to the detected bearing and motion of the mobile agent 100b (target). The controlled agent dynamics 3015 in the STMR loop refers to the physical motion response of the mobile agent 100a resulting from actuator commands provided by the processor 208 to the actuator 204, governed by the agent's kinematic or dynamic model.Stability Analysis of the Multi-Agent SystemBi-agent stability
[0337] For stability analysis of the multi-agent feedback system, we consider differentialdrive ground vehicles set at a constant forward speed v, following the non-holonomic constraint, whose kinematics can be defined for 2D horizontal planar motion aswhere utis the feedback control input given in Eqn. 3.3.4 and vt> 0 is the fixed forward speed.
[0338] FIG. 31 illustrates the geometry of the bi-agent problem.
[0339] First consider a bi-agent system with only two agents interacting with each other. Let the position error, defined as the distance between the mobile agents, be given asx = xa- xby = ya-yb (3.3.6)
[0340] Let 0 be the angle of the line-of-sight (LOS) with respect to the x axis, then from FIG.31.x = rcos0ty = rsin0t(3.3.7)
[0341] Differentiating error x and y with respect to time yieldsx = rcos0t— rsin0t0t= vacos0a— xby = rsin0t+ rcos0t0t= vasin0a— yb(3.3.8)
[0342] Rewriting the equations for r and 0 yieldsr = vacos(0a- 0t) - (xbcos(0t) + ybsin0t),(3.3.9)0 = i(vasin(0a- 0t) + (%bsin(0t) - ybcos0t)).
[0343] Substituting xb= vbcos0band yb= vbsin0bin Eqn. 3.3.9 simplifies to(3.3.10)
[0344] The feedback law for horizontal planar motion is given by
[0345] where K is the feedback gain constant and y is the azimuth angle. Elevation angle, P = 0 for horizontal planar motion.
[0346] From FIG. 31, the azimuth angles yaand ybcan be defined asYa ~ t a> (3.3.12)yb= 7l + 0b~ et.
[0347] Since both the mobile agents have the same forward speed, va= vb= v. Let the nearness function be defined as / z = 1 / r. For notational convenience, define a = vji.
[0348] Using Eqns. 3.3.10 and 3.3.11, and differentiating Eqn. 3.3.12, the error dynamics with state variables yaand ybcan be expressed in the polar coordinate as(3.3.13)
[0349] Assume only one input to the system and let Kb= 0, making the motion of point B a straight line. Then, linearizing the above system about the origin gives(3.3.14)
[0350] From the characteristic equation of this system, it can easily be seen that the origin is exponentially stable for Ka> 0.Multi-agent stability
[0351] Now, to analyze the multi-agent switching system with N agents, Theorem 2 from is considered which describes stability of switched systems with multiple equilibria under disturbances. The theorem predicts bounded trajectories for input-to-state stable (ISS) subsystems under average dwell time bounds. The average dwell time Nais defined in terms of the number of switches Na(t, t) G Z+over the interval [t, t) caswhere No> 0 is a finite constant. The bounds for average dwell time is given bywhere Na> 0 represents the lower bound on the average dwell time of the system. For our system, we define Nafor any constant e in the open interval (0,1) aswhere A > 0 and n(k') > 1 are defined where p represents the maximum sudden growth in error that occurs during a system switch, and X represents the continuous exponential stabilization rate while the system dwells in a single mode.Simulation Study
[0352] Simulation of the multi-agent switched system in FIG. 30 under average dwell time bound from Eqn. 3.3.16 was implemented in Matlab. A total of N = 20 homogeneous agents are considered in the 2D planar motion simulation environment and assumed to be following the kinematics of ground vehicles described in Eqn. 3.3.5. This results in N — 1 identical subsystems that the STMD output can switch between for each mobile agent. The simplified simulation assumes sparse environment with no background or obstacles, and no explicit collision avoidance algorithm in place (outside of any given by the feedback law). All of the mobile agents are set at the same fixed forward speed of vt= 10 cm / s with some arbitrary initial positions and orientations. Gains Ktare arbitrarily set to 0.1. Parameters to determine the minimum average dwell time are set arbitrarily to: p(fc) = 10, A = 1 and e = 0.3. Measurement of optic flow and STMD output uses the idealized equations from Eqn.4.2.1 and Eqn. 3.3.1 respectively. Target switching for each mobile agent is allowed only if the average dwell time bound is satisfied. Each simulation is run for 50 seconds.Experimental Study
[0353] As shown in FIGS. 19A— 19C, for real-time experimental implementation on ground robots, five Ground-based mobile agents 100 equipped with panoramic cameras 1905 and running the decentralized feedback controller onboard are used, starting from arbitrary initial conditions that were kept constant for all the experiments. The experiment arena uses plain white boundary walls for low visual texture to ensure only the swarm agents' motions are generating the measured optic flow signal. The Ground-based mobile agents are operated byRaspberry Pi 3b running Raspbian OS. Python, C++, and ROS Kinetic are used to implement the onboard feedback system processes, including onboard image acquisition and processing, feedback control calculation and implementation. The mobile agents have a maximum linear velocity of 22 cm / s and maximum rotational velocity of 2.84 rad / s. No explicit collision avoidance algorithm is implemented on the mobile agents 100. The experimental arena spans 8 x 12 ft. The camera used is Picam360 with a vertical FOV of 220° and a horizontal FOV of 360°. It captures 640x480 pixel images at 20 fps. The raw, circular image is then distortion corrected into a panoramic image using image processing. The camera 1905 is mounted face down to ensure the target agents occupied a large FOV in the dewarped images. A single horizontal line from the input image contains all the target agents and is used to perform onboard optic flow calculation. Optic flow calculation from the camera image input is performed using the Gunnar Farneback algorithm using the opencv_apps ROS package. The Gunnar Farneback algorithm calculates dense optic flow at fixed points in the visual input based on pixel intensity changes between successive image frames.
[0354] A remote desktop running Linux-based Ubuntu 16.4 initializes all the mobile agents 100 at the start of the experiment. Each experimental trial consisted of 15 seconds.
[0355] Two cases have been explored in this setup for comparison: (a) pure pursuit guidance law STMR, which uses the feedback rule defined in Eqn. 3.3.4, and (b) motion camouflage guidance law STMR, which uses the feedback rule for motion camouflage, where an agent continuously nullifies the transverse component of its relative velocity to ensure its line of sight to the target remains constant. Both cases use STM D-based sensory model for optic flow processing, given in Eqn. 3.3.1. The scalar feedback gains Ktwere chosen arbitrarily, and the average dwell time was chosen to be 1.5s, based on prior stability analysis. All the ground-based mobile agents, i.e., an example of mobile agents 100, were set with a fixed forward speed of vt = 11 cm / s. Gains Ktand average-dwell time were also the same for all agents. The initial conditions for all agents were chosen arbitrarily and kept constants for all trials of both the cases.Comparison Study
[0356] For comparison of the STMR framework with existing multi-agent systems, three other multi-agent systems are considered: the Vicsek model, the Cucker-Smale (C-S) model and the multi-agent optomotor response (MA-WFI) model developed previously by the authors that uses wide-field integration (WFI) of optic flow signals for regulating an agent'smotion relative to its neighbors. The Vicsek model updates agent orientation based on the average orientation of mobile agents in its neighborhood, defined by a radius of interaction. The C-S model assumes inter-agent interaction occurs between all the mobile agents in the environment, regardless of their inter-agent distances, and the mobile agents update velocity based on a weighted function of the average velocity of all the other agents. The Vicsek model achieves orientation-matching whereas C-S model does velocity-matching. The MA-WFI model is designed to be able to do both orientation-matching and velocity-matching for a controlled-agent in an established formation of other agents based on the choice of basis functions.
[0357] Two different environments are considered for comparison of feedback systems. For the first case, the MA-WFI simulation environment is used, where only one agent is controlled among other agents in an established formation. The controlled agent has an initial orientation offset, while the rest starts with no offset. The simulation is run four times, once for each of the different feedback control systems. For the second case of comparison, all agents are being controlled by the four different control systems starting from the same initial conditions as in the first case. The C-S agents have varying initial velocity, the other models have the same fixed forward speed for all agents during the entire simulation.
[0358] For further comparison of the STMR framework, we study the connectivity of a system of 50 agents governed by the four different multi-agent systems used earlier: STMR, MA-WFI, C-S and Vicsek. For this study, the mobile agents are converted to nodes (V) of a weighted graph Q = (V, E, W') and there is an edge (E) between a pair of nodes only if the two agents interact with each other. Edge weights (W) are determined based on the feedback rules that the mobile agents use. Algebraic connectivity (also called the Fiedler eigenvalue) is defined as the second-smallest eigenvalue of the Laplacian matrix of Q and is a measure of how well connected the mobile agents are among each other in a multi-agent system.Results and DiscussionResultsSimulation results
[0359] Simulation results for six different initial conditions are presented in FIGS. 32A-32F. The markers represent the final position of the mobile agents while the solid lines represent their trajectories. Collision avoidance is not designed into the feedback rule, so collision between mobile agents is observed in some instances. FIGS. 33A-33F show all the mobileagents' orientationsover the simulation period. The mobile agents show general directional consensus and bounded for each of the different initial conditions.
[0360] FIG. 34A presents the switching signal plot with and without the average dwell time bounds. The plot indicates which agents are 'tracked' over time by an arbitrary agent. Each mobile agent 'tracks' one other agent based on the STMD-output (Eqn. 3.3.1). FIG. 34B shows the average dwell time plot of an agent. The minimum average dwell time based on Eqn. 3.3.16 is also plotted. It isclearthat switching between 'tracked' agents occurs only when the average dwell time for that agent is above the minimum specified. After each switch, the average dwell time decreases as expected from Eqn.3.3.15, resulting in the sawtooth-like shape of the plots.
[0361] FIGS. 35A-35F show the mobile agent trajectories for simulation of the system without the average dwell time bounds, allowing the mobile agents to switch instantaneously to whichever neighboring agent has the highest magnitude of optic flow. Initial conditions for the six cases were kept the same as those in FIGS. 32A-32F for comparison. The mobile agents still show general directional consensus over time, but one observes fragmentation into smaller groups of mobile agents in some cases (i.e. FIG. 35C).Experimental results
[0362] Experimental results for the two feedback rules are shown in FIGS.36A and 38B. Blue lines represent mean agent trajectories, while the grey areas represent the variance over 15 trials for the same initial condition. It can be seen clearly that the mobile agents converge towards each other for pure pursuit STMR feedback in FIG. 38A. Collision occurs at least for a couple of mobile agents since no collision avoidance algorithm is being implemented. Unlike within the simulation, collision has a greater effect in the experiments as ground vehicles continue to interfere after collision. Increasing initial agent separation and reducing the experiment duration to a relatively shorter time (15s) compared to the simulation (50s) allowed the multi-agent system to show convergence behaviors while reducing the effects of collision-induced behavior. The present disclosure does not analyze the effects and aftermath of collision among agents.
[0363] For the case of using motion camouflage STMR feedback, results shown in FIG. 36B, the mobile agents do not converge or collide at all, and do not seem to deviate much from their trajectories. Agents' group motion demonstrates higher polarization and more consistent spatial separation.Discussion
[0364] The implementation results of the feedback control system proposed in FIG. 30 suggest that group motion can be achieved using the small-target motion reactive (STMR) framework. The mobile agents' states are bounded over time (FIGS. 32A-33F) and group motion is achieved. The average dwell time constraint ensures switching between agents is sufficiently slow to ensure boundedness, but the group shows general directional consensus without the constraint as well (FIGS 35A-35F). In this framework, each mobile agent tracks only one other agent in the environment, instead of tracking all or a large subset of mobile agents as in other traditional multi-agent systems. This means the system is easily scalable to large groups of mobile agents without the need for increased sensory capabilities or communication bandwidth.
[0365] The simulation results of the multi-agent system suggests that the addition of average dwell time bounds on the switching system ensures stability in terms of bounded states and prevents high frequency switching between the subsystems. Trajectories without the average-dwell time bound (FIGS. 35A-35F) suggest very high frequency switching can lead to factions splitting from the group. Parameters such as e and n(k') can be used to tune the average dwell time bound to allow higher switching frequency at the cost of the trajectories converging to a bigger compact set of solutions. The trade-off is between switching frequency and robustness of the. Several mechanisms for attention-like processes and selective attentional capabilities have been discovered from prior studies of insect attention. The STMD neurons in insects such as dragonflies shows similar behavior by tracking only one target when multiple targets are presented, even "locking onto" a low contrast target in spite of the introduction of a high-contrast distractor. This "lock-on" mechanism reduces the number of switches between targets, but does not eliminate them completely. While the "lock-on" phenomenon is claimed to reduce confusion during prey selection in a swarm, the results from this study may be used to show that it may also be applicable to cohesive motion coordination.
[0366] FIG 37 discloses a single feedback-controlled mobile agent surrounded by non-reactive neighbors with established formation; comparison of STMR feedback control system with the optomotor feedback control system from previously published work by Applicant, the Vicsek model and the Cucker-Smale model.
[0367] From the comparison study in FIGS. 38A-38C, the STMR agent trajectories show a more unstructured behavior over time compared to the other models, due to the fact that the stability analysis only guarantees boundedness for the STMR agents instead of convergence. Factions of mobile agents can split and rejoin (FIG. 32F), or agents show "milling" motion that does not eventually settle down to any particular group formation. This propagation of 'erratic' behavior is different than traditional swarm models found in literature that use attraction, repulsion and / or state alignment for stability. However, the lower polarization levels observed from STMR agents can fairly resemble the seemingly less structured movement of insects such as honeybees in swarm.
[0368] Conventional swarming approaches often assume continuous large bandwidth of information sharing among agents (FIGS. 39A and 39B), and low-error communication systems which are not easily scalable or implementable in large groups. These approaches do not account for the neural / computational limit of the biological animals that it models, nor of the hardware that will be used for implementation. Since STMR agents only need information from the largest optic flow signal producer, individual agents can be disconnected from the rest of the N — 2 agents and still have group motion, as this study suggests (FIGS. 39A and 39B). Since the information shared in conventional approaches is to provide convergence, the connectivity is much higher at the start of the simulation and decreases over time as agents converge to an alignment consensus (FIG. 39A). Instead of having very high connectivity at the start of the simulation to prioritize convergence, STMR spreads the attentional workload over time to reach bounded trajectories instead, and shows consensus over group directionality (FIG. 39B). Low connectivity could sometimes lead to factions splitting off, but results suggest that these factions eventually rejoin, given the average dwell time bounds are enforced. See FIGS. 32D and 32F.
[0369] The STMD neurons in predatory insects such as dragonflies predominantly aid in pursuer-evader scenarios, STMD neurons have been found in non-predatory insects as well. This study proposes a method of using pairwise interaction between agents for collective motion in multi-agent systems and there is precedent to believe that this might also be the case in animals. Flocking starlings interact with 6-7 neighbors, while honeybees in convective group milling interact with 1-2 neighbors. A similar behavior as observed from STMR agents, such as faction splitting and rejoining resulting in robust cohesion, is also observed in flockingstarlings to avoid predators, as the swarm constantly changes its density and structure and yet no agent is isolated from the group.
[0370] FIGS 38A-38C disclose a collection of feedback-controlled agents all reactive to each other; comparison of the STMR feedback control system with the optomotor feedback control system from previously published work by Applicant, the Vicsek model, and the Cucker-Smale model.Discussion of assumptions and implications
[0371] This model assumes the small-target motion detectors can reliably suppress background motion, which is generally slow-moving and further away, generating low frequency optic flow signals, similar to how STMD neurons in dragonflies respond to background motion. In contrast, the fast-moving small-target agents closer to the observer are assumed to provide high frequency optic flow signals that are used as input for the system. The model also assumes STMD neurons are perfect sensors with no temporal or frequency dependence, resulting in zero time delay and signal amplification over a wide range of frequencies. This assumption allows for investigating the effect of concise STMD-type neural sensing mechanisms in multi-agent systems in idealized conditions before further analysis of the sensory model including temporal dynamics.
[0372] No exclusive collision avoidance algorithm is implemented for the STMR system, and the results suggest the STMR feedback rule does not provide repulsion in close range. In simulation, the mobile agents were allowed to overlap and continue in their trajectories, while in experiment the mobile agents collided and moved together after collision. Separating the effect of collision avoidance ensured the effect of STMD-based sensing in group cohesion could be analyzed in depth.
[0373] The Veer and Poulakakis theorem used for stability analysis of the switched system provides a bound on the state trajectories for multiple equilibrium of the sub-systems. The behaviors discussed thus require some switching rate limit provided by the average dwell time constraint, thereby ensuring boundedness of mobile agent states. Similar behavior can be seen without the constraint (FIGS. 35A-35F) but the mobile agents are then more prone to splitting into multiple groups. Attention in insects have been well studied and several mechanisms identified for attention-like processes. Insects such as Drosophila have also been found to have attention-like mechanisms where they can shift the focus of attention (FoA) internally, with the time-span between each shift of FoA being ls-4s, values comparable tothe average dwell time bounds used in the bio-inspired feedback model designed in this study. In addition, insect attention has multiple definitions and unifying them is outside the scope of this study. Instead, we use the ad-hoc term attentional work to describe the area under the union of graphs (FIG. 39), to provide a quantitative comparison of the total information shared (or "informational work done") by the connected agents.
[0374] The comparison and connectivity plots among different multi-agent systems in FIGS.37-39B are one instance of simulation amid numerous other possible trajectories based on the gains and tunable parameters chosen for each system. Instantaneous connectivity values are dependent on the gains used for convergence, but the trend remains similarto that shown in FIGS. 39A and 39B. Different gains might affect the time it takes for attentional work to reach equality for STMR and the other multi-agent systems, meaning the overall attentional work of said systems are equal overtime, but does not necessarily affect the existence of such a finite time.Discussion
[0375] The implementation results of the feedback control system proposed in FIG. 30 suggest that group motion can be achieved using the small-target motion reactive (STMR) framework. The mobile agents' states are bounded overtime (FIGS. 32A-32F and FIGS. 33A-33F) and group motion is achieved. The average dwell time constraint ensures switching between agents is sufficiently slow to ensure boundedness, but the group shows general directional consensus without the constraint as well (FIGS. 35A-35F). In this framework, each mobile agent tracks only one other agent in the environment, instead of trackingall ora large subset of mobile agents as in other traditional multi-agent systems. This means the system is easily scalable to large groups of mobile agents without the need for increased sensory capabilities or communication bandwidth.
[0376] The simulation results of the multi-agent system suggests that the addition of average dwell time bounds on the switching system ensures stability in terms of bounded states and prevents high frequency switching between the subsystems. Trajectories without the average-dwell time bound (FIGS. 35A - 35F) suggest very high frequency switching can lead to factions splitting from the group. Parameters such as e and n(k') can be used to tune the average dwell time bound to allow higher switching frequency at the cost of the trajectories converging to a bigger compact set of solutions. The trade-off is between switching frequency and robustness of the system. Several mechanisms for attention-likeprocesses and selective attentional capabilities have been discovered from studies of insect attention. The STMD neurons in insects such as dragonflies shows similar behavior by tracking only one target when multiple targets are presented, even "locking onto" a low-contrast target in spite of the introduction of a high-contrast distractor. This "lock-on" mechanism reduces the number of switches between targets, but does not eliminate them completely. While the "lock-on" phenomenon is claimed to reduce confusion during prey selection in a swarm, the results from this study may be used to show that it may also be applicable to cohesive motion coordination.
[0377] From the comparison study in FIGS. 38A-38C, the STMR agent trajectories show a more unstructured behavior over time compared to the other models, due to the fact that the stability analysis only guarantees boundedness for the STMR agents instead of convergence. Factions of mobile agents can split and rejoin (i.e. FIG. 32F), or agents show "milling" motion that does not eventually settle down to any particular group formation. This propagation of 'erratic' behavior is different than traditional swarm models found in literature that use attraction, repulsion and / or state alignment for stability. However, the lower polarization levels observed from STMR agents can fairly resemble the seemingly less structured movement of insects such as honeybees in swarm.
[0378] Conventional swarming approaches often assume continuous large bandwidth of information sharing among agents (FIGS. 39A and 39B), and low-error communication systems which are not easily scalable or implementable in large groups. These approaches do not account for the neural / computational limit of the biological animals that it models, nor of the hardware that will be used for implementation. Since STMR agents only need information from the largest optic flow signal producer, individual agents can be disconnected from the rest of the N — 2 agents and still have group motion, as this study suggests (FIG. 39A and 39B). Since the information shared in conventional approaches is to provide convergence, the connectivity is much higher at the start of the simulation and decreases over time as agents converge to an alignment consensus (FIG. 39A). Instead of having very high connectivity at the start of the simulation to prioritize convergence, STMR spreads the attentional workload over time to reach bounded trajectories instead, and shows consensus over group directionality (FIG. 39B). Low connectivity could sometimes lead to factions splitting off, but results suggest that these factions eventually rejoin, given the average dwell time bounds are enforced (FIGS. 32D and 32F).
[0379] The STMD neurons in predatory insects such as dragonflies predominantly aid in pursuer-evader scenarios, STMD neurons have been found in non-predatory insects as well. This study proposes a method of using pairwise interaction between agents for collective motion in multi-agent systems and there is precedent to believe that this might also be the case in animals. Flocking starlings interact with 6-7 neighbors, while honeybees in convective group milling interact with 1-2 neighbors. A similar behavior is observed from STMR agents, such as faction splitting and rejoining resulting in robust cohesion, is also observed in flocking starlings to avoid predators, as the swarm constantly changes its density and structure and yet no agent is isolated from the group.Discussion of assumptions and implications
[0380] This model assumes the small-target motion detectors can reliably suppress background motion, which is generally slow-moving and further away, generating low frequency optic flow signals, similar to how STMD neurons in dragonflies respond to background motion. In contrast, the fast-moving small-target agents closer to the observer are assumed to provide high frequency optic flow signals that are used as input for the system. The model also assumes STMD neurons are perfect sensors with no temporal or frequency dependence, resulting in zero-time delay and signal amplification over a wide range of frequencies. This assumption allows for investigating the effect of concise STMD-type neural sensing mechanisms in multi-agent systems in idealized conditions before further analysis of the sensory model including temporal dynamics.
[0381] In the present embodiment, no exclusive collision avoidance algorithm was implemented for the STMR system, and the results suggest the STMR feedback rule does not provide repulsion in close range. In simulation, the mobile agents 100 were allowed to overlap and continue in their trajectories, while in experiment the mobile agents 100 collided and moved together after collision. Separating the effect of collision avoidance ensured the effect of STMD-based sensing in group cohesion could be analyzed in depth.
[0382] The Veer and Poulakakis theorem used for stability analysis of the switched system provides a bound on the state trajectories for multiple equilibrium of the sub-systems. The behaviors discussed thus require some switch rate limit provided by the average dwell time constraint. Similar behavior can be seen without the constraint (FIGS.35A-35F) but the mobile agents are then more prone to splitting into multiple groups. Attention in insects have been well studied and several mechanisms identified for attention-like processes. Insects such asDrosophila have also been found to have attention-like mechanisms where they can shift the focus of attention (FoA) internally, with the time-span between each shift of FoA being ls-4s, values comparable to the average dwell time bounds used in the bio-inspired feedback model designed in this disclosure. In addition, insect attention has multiple definitions and unifying them is outside the scope of this disclosure. Instead, we use the ad-hoc term attentional work to describe the area under the union of graphs (FIG. 39B), to provide a quantitative comparison of the total information shared (or "informational work done") by the connected agents.
[0383] The comparison and connectivity plots among different multi-agent systems in FIGS.39-41B are one instance of simulation amid numerous other possible trajectories based on the gains and tunable parameters chosen for each system. Instantaneous connectivity values are dependent on the gains used for convergence, but the trend remains similarto that shown in FIGS. 39A and 39B. Different gains might affect the time it takes for attentional work to reach equality for STMR and the other multi-agent systems, meaning the overall attentional work of said systems are equal overtime, but does not necessarily affect the existence of such a finite time.Summary and conclusions
[0384] This study introduces the small-target motion reactive (STMR) approach to insect swarming by designing an engineering model of the STMD neurons found in the lobula of insects such as hoverflies and dragonflies. The two major outputs from this group of motion detection neurons are the highest magnitude of optic flow and the bearing angle corresponding to the moving agent generating the highest magnitude of optic flow. Assuming an idealized case of the STMD neuron model, a multi-agent switched feedback system framework (STMR) is developed. Switched feedback system refers to a system where the selected (target) neighbor changes over time. Stability analysis of the multi-agent system involves bi-agent stability of each subsystem, which extends to state boundedness via the Veer-Poula kakis theorem for switched systems. An average dwell time constraint ensures the mobile agent states are bounded over time by ensuring that switching between alternating agents is sufficiently slow, thereby insuring system stability. The feedback control system is implemented in simulation and on ground vehicles for validation, resulting in agent trajectories that demonstrate group cohesion and general directional consensus. Comparison with other multi-agent systems such as Vicsek and Cucker-Smale models suggest that the lowconnectivity network with an agent only tracking one other agent may still result in group motion comparable to conventional swarm algorithms but with consistently very low sensory computational requirements. Overall, STMR represents a bio-inspired strategy that could provide a similar function to traditional engineered swarm approaches while accounting for low computational capabilities of insect visuomotor systems by spreading the processing workload over time and limiting the amount of instantaneous individual "attention" requirements to the surrounding.IV. PERSUASIVE LEADERSHIPIntroduction
[0385] In one of the most prevalent examples of insect swarming, only a small percentage of bees (less than 5%) in a honeybee swarm are aware of the location of the new nesting site during migration of the entire colony, having scouted the location. This small collection of high speed, informed "streaker" bees provides guidance to direct the remainder of the swarm to the nest site. Insect swarm study has identified the presence of these streaker bees, but the underlying mechanism on how the information is exchanged between the small number of informed bees and the large number of uninformed bees is not well understood. Though some studies have proposed mathematical models for swarm guidance based on streaker bees, there have been few models based on insect visuomotor system and its role in group motion.
[0386] Insects such as bees show great flight maneuverability during group motion, using only a small collection of neurons in their visuomotor system. Recent study has shown how visual feedback based on insect visual pathway can participate in swarm motion. The models use optic flow signal from the surrounding to extract relevant motion cues. This study explores whether the streaker bees are able to influence and 'inform' the swarm in a similar way using only optic flow based feedback framework.
[0387] Studies of bees have documented how only a small collection of bees visits a new nesting site and is therefore aware of its location before the entire colony migrates to it. There have been many hypotheses on how the informed bees transfer information to the swarm, one of them being proposed by Lindauer in 1955 termed the streaker bee hypothesis that suggests the informed bees make high speed flights, or in other words, streak through the swarm towards the direction of the new nest, and then slowly fall back to the back of theswarm in order to guide the uninformed bees. This idea has been tested both by experimental observations and by mathematical models.
[0388] Beekman et al. recorded swarm flight data and observed a small minority of bees flying through the swarm at high speeds, while the rest of the swarm moved at lower speeds. Schultz et al. recorded and tracked individual bees from swarm flight videos to show that there were two distinct groups of bees flying at two different speeds during migration. Latty et al. exposed flying honeybee swarm to a group of fast-moving bees headed towards the wrong direction, and found that the conflicting information resulted in swarm segregation and failure to reach the hive, further supporting the streaker bee hypothesis.
[0389] Previous work quantifying in-flight insect sensing and feedback has identified the importance of feedback paths using neurons specialized in responding to temporal gradients of retinal image intensity, or optic flow. The visual neurons' sensitivity patterns to optic flow are consistent with a feedback approach that extracts relative motions. For example, opticflow relevant feedback plays a major role in forward speed regulation and centering response through tunnels with varying spatio-temporal frequencies of moving grated boundary walls, maintenance of constant flight speed through varying tunnel widths, regulating flight height, and in visual path integration, findings which have been used to develop biologically relevant feedback models that use lobula plate tangential cell basis functions or their engineered analogues. This disclosure uses one such bio-inspired multi-agent feedback model developed by Bi Ila h et al. to study the effect of streaker bees on the swarm.
[0390] This portion of the disclosure considers the two bio-inspired multi-agent feedback control systems developed previously in the disclosure and discusses the implementation of the modified system with streaker agents in simulation, then presents the simulation results and discusses the findings and outcomes from this study and their implications.Methods and Approach
[0391] As previously stated regarding FIG. 1: in this section, a detailed approach to the study is presented, which is to design a multi-agent environment with two types of mobile agents. The first type comprises standard one or more mobile agents 100 acting as 'swarm1agents 110 that follow a bio-inspired visual feedback system. The second type comprises mobile agents configured as informed agents 105. Some of these informed agents 105 may be referred to hereinafter as 'streaker agents', which follow a predefined constant, high-speed trajectory without relying on neighborhood feedback rules.Bio-inspired multi-agent systemsMA-WFI multi-agent system
[0392] Recall the multi-agent wide-field integration (MA-WFI) of optic flow feedback control system disclosed previously. The feedback control framework used for the swarm agents in this study is based on an existing formulation with experimental results available to represent idealized sensors. In wide-field integration (WFI), optic flow measurements are used as feedback to navigate the controlled agent among neighboring agents and maintain a swarm formation during motion, as modeled in FIG. 7. The simplified model assumes horizontal planar motion where the neighboring agents already have an established formation while the singular controlled agent regulates its trajectory for some initial condition, as shown in FIG.5. Boundary walls are not considered in designing the feedback control algorithm, neither are the presence of obstacles in the environment.
[0393] The optic flow detected by the mobile agents due to its own motion and the motion of one neighboring agent can be written as a function of both of their stateswhere the mobile agents' own states are qb= xb,yb,0 and qb= (xb,yb, 0), neighboring agent states are qt= (xt,yt,0t) and qt= (xt,yt,0t), y is the bearing angle between the mobile agents and the nearness function, n, can be generally expressed as" = i?h’(4-2-21where rbtis a vector connecting the two agents. Assuming no collision between agents at any given time, rbt= 0 and n G (0, oo).
[0394] Wide-field integration of optic flow is performed by the lobula plate tangential cells of the visuomotor system of insects to extract relevant information about its surroundings from the input optic flow pattern and can be modeled as a spatial inner product in a function space L2S2,^2) aswhere Ft(y) is the set of weight functions that represents the cells' sensitivity to various optic flow patterns.STMR multi-agent system
[0395] Recall the small-target motion reactive (STMR) multi-agent feedback control system presented above. The small-target motion detector (STMD) neurons found in insects such as dragonflies are mathematically modeled as maxima (max) and argument of maxima (arg max) operators acting on the optic flow output from its preceding neural pathway that identifies the maximum magnitude of optic flow and its corresponding angular position from the entire visual field of view. Mathematically, this operation may be expressed asystmd,i = rnax(Q)(4-2.4)ystmd,2 = argmax(Q),rwhere Q is the optic flow signal and y is the viewing angle. The block diagram for the resulting switching system is shown FIG. 31. For the horizontal planar motion considered in this study, y spans between [— n,n] and the idealized optic flow equation for multi-agent environment can be expressed as in Eqn. 2.2.1. This formulation relies on neighboring agents and does not model environmental effects such as boundary walls or obstacles.
[0396] Simulation result for five agents using STMR feedback control are shown in FIG. 42. Stability analysis of this system ensures bounded trajectories under average dwell time constraints for the switching signal, which has also been implemented for this study.SimulationMA-WFI feedback control system
[0397] In this study, MA-WFI feedback is used as the feedback control input for all agents, rather than a singular agent. No established formation of mobile agents are assumed and thus group consensus of such a system has not yet been guaranteed. Each swarm agent calculates the idealized optic flow (Eqn. 4.2.1) generated by the motion of all the other agents, followed by the wide-field integration of optic flow (Eqn.4.2.3). Simulation of the proposed feedback system is then computed in Matlab. A total of N = 6 swarm agents and M = 2 streaker agents are considered for the simulation environment and the mobile agents are assumed to be particles following the kinematics of differential drive ground vehicles with the non-holonomic constraintwhere utis the ith system's control input, i = {1,2,..., Af} and v = x^ is the fixed forward speed of the swarm agents. Ground robots are often a more suitable platform for experimental testing of multi-agent systems in controlled laboratory environments. Using a similar agent model in simulation makes the results more comparable to actual experiments.
[0398] Two cases are considered for this simulation. The first case implements the nominal MA-WFI feedback control system without the presence of streaker agents. The second case considers the same number of swarm agents in the presence of two high speed streaker agents following a fixed trajectory. Both simulations assume sparse environment with no background. Optic flow is generated only by the mobile agents' motion. Collision avoidance is not set up for this simulation. All swarm agents are set at the same fixed speed of 8 cm / s with some random arbitrary initial positions and orientations. Streaker agents are set at double the speed of swarm agents at 16 cm / s with some random arbitrary initial position and no orientation offset. To persistently understand the influence of streaker agents on the swarm agents, the streaker agents are set to 'come back' once the fast moving streaker agents move 'too far away' from the swarm agents. Parameters that determine when and where to 'reset' the streaker agents' positions were kept constant throughout the simulation. Gains K and K2are arbitrarily chosen from the set of allowable gain values, and set to be —0.5 and —0.2 respectively. Measurement of optic flow and WFI output uses the idealized equations from Eqn. 2.2.1 and Eqn. 4.2.3 respectively. All simulations were run for 50 seconds.STMR feedback control system
[0399] In this section, the effect of introducing high-speed 'streaker' agents following predefined trajectories among a group of low-speed 'swarm' agents using STMR feedback for navigation is observed. Each swarm agent calculates the idealized optic flow (Eqn. 4.2.1) generated by the motion of all the other agents in the environment, followed by the STMD output (Eqn. 4.2.4). Pure pursuit is used as the feedback law to close the loop, given bywhere K is some arbitrary gain. Since the target's magnitude of optic flow is not required for pure pursuit algorithm, ystma,i remains unused for this simulation.
[0400] Simulation of the proposed feedback system may be computed in Matlab. A total of N = 6 swarm agents and M = 2 streaker agents are considered for the simulation environment and the mobile agents are assumed to be particles following the kinematics of differential drive ground vehicles with the non-holonomic constraint given in Eqn. 4.2.5. For streaker agents, u = 0 and v = xst, where xst> x^. Ground robots are a convenient multiagent experimental platform for controlled laboratory testing, and simulations having agent model similitude improve the results' agreement with experimental work.
[0401] Two cases are considered for this simulation. The first case implements the nominal STMD based multi-agent feedback control system without the presence of streaker agents and represents baseline motion. The second case considers the same swarm agents and initial conditions in the presence of two high speed streaker agents following a fixed trajectory. Both simulations assume a sparse environment with no background, thus optic flow is generated only by the mobile agents' motion and explicit collision avoidance is not implemented for this simulation. All swarm agents are given the same fixed speed (8 cm / s) with randomly distributed initial positions and orientations. Streaker agents have triple the speed of swarm agents (24 cm / s) with random arbitrary initial position at the edge of the swarm and an assigned orientation (n / 5 rads). To persistently understand the influence of streaker agents on the swarm agents, the streaker agents are set to re-spawn closer to the swarm once the fast moving streaker agents have traveled beyond the swarm agents. Parameters that determine when and where to reset the streaker agents' positions were defined by the interagent distancesand were kept constantthroughout the simulation. The gain K was arbitrarily chosen from the set of allowable gain values to be 0.1. Measurement of optic flow and STMD output uses the idealized equations from Eqn. 4.2.1 and Eqn. 4.2.4 respectively. The average dwell time constraint was set arbitrarily to Is, with target switching based on the maximum optic flow value only allowed once this average dwell time bound is satisfied. Each simulation case was run for 50 seconds.Results and Discussion for the MA-WFI systemSimulation results
[0402] Simulation results for both the cases and six different initial conditions of the swarm agents are presented in FIGS.44A-44F. The markers represent the final position of the mobileagents while the solid and dotted lines represent their trajectories. The blue and gray lines represent the same simulation with and without the influence of streaker agents respectively. Yellow line shows the streaker agents' fixed trajectories, kept constant for all instances. In two instances (FIGS. 44B and 44C), the mobile agents show group orientation alignment by MA-WFI feedback without the influence of streaker agents and the nominal feedback system is claimed to be stable. The nominal feedback system without the streaker agents is unstable for the other instances shown. For the modified feedback system with streaker agents on either side at higher speeds, most swarm agents change direction towards the motion of the streaker agents in all the instances. Modified feedback system thus has greater number of stable agents for all initial conditions. Some agents (FIG. 44D) align parallel to the motion of the streaker agents, but move in the opposite direction, while some agents (FIGS. 44D and 44E) show no visible change in trajectory in the presence of streaker agents.
[0403] FIGS.45A and 45B show the average orientation, 0avg, of the swarm agents for both the cases for two instances of initial conditions, (FIGS. 44C and 44F). It is clear from the plots that the average swarm heading is influenced by the streaker agents to be close to their own orientation of 0b= 0. Oscillation in FIG.45B is observed due to the streaker agents being too far away from the swarm agents, right before being 'reset', which causes one of the swarm agents to oscillate. This implies a region of influence of the streaker agents on the swarm agents. Implications of simulation results are discussed in the next section.Discussion
[0404] The simulation results of the MA-WFI feedback control system in presence of high speed 'streaker' agents suggest that the general orientation of mobile agents during group motion can be influenced in part by the presence of high speed, 'informed' agents, as evident in nature. The MA-WFI feedback controller ideally corrects agents' orientation until the optic flow signal is balanced on either side. This can explain the alignment of some agents parallel to the streaker agents' trajectories but in the opposite direction (FIGS. 44A and 44D). When the initial orientation 0O< — y or 0O> y, the mobile agents can reach optimum WFI output by moving in the opposite direction, 0b= —0*, since the optic flow signal will still be balanced.
[0405] The plots in FIGS. 45A and 45B show average group orientation varies over time for the first case, whereas the streaker agents influence a more constant average grouporientation over time, which is also close to the streaker agents' orientation. The presence of swarm agents that are moving in parallel but opposite direction might be causing the offset in the average swarm orientation and the streaker agent orientation.
[0406] The mobile agents that keep the same trajectory with and without the streaker agents leads us to conclude that there is a region of influence somewhere in between the streaker agents. Any agent starting outside that region of influence does not react to the motion of the streaker agents. From the results shown here (FIGS. 44B and 44D), the mobile agents starting at the same horizontal position as the streaker agents are less likely to be influenced by it. Region of influence is also attributed to the oscillation of one agent in FIGS.44F, resulting in an oscillation of the average orientation in FIGS. 45B. The oscillation stops once the streaker agents are 'reset' closer to the swarm agents. This suggests the inter-agent distance between the swarm agents and the streaker agents could be a parameter that determines this region of influence. Detailed study of the region of influence in future will help us explain the role of streaker agents in swarm motion much better.Results and Discussion for the STMR systemSimulation results
[0407] Simulation results for six different initial conditions of the swarm agents for the two cases are presented in FIGS. 41A-41F. The markers represent the final position of the mobile agents while the solid lines indicate their trajectories. The blue and gray lines represent the same simulation with and without the presence of streaker agents respectively, while yellow lines indicate the streaker agents' predefined trajectories, kept constant for all instances.
[0408] For all instances presented, the swarm agents show group cohesion and similar direction of motion by STMR feedback even without the presence of streaker agents. Multiagent group motion is established for the baseline case but without any average group orientation guarantees. For the modified multi-agent system with the 'informed' streaker agents, the swarm agents still establish group motion and cohesion but the mobile agents also change direction towards the motion of the streaker agents. FIGS.43B and 43C show an increase in the inter-agent distances in presence of the streaker agents, likely in an attempt to track them, but the increased inter-agent distance does not lead to segregation. Most of the mobile agents can be seen 'chasing' only one of the streaker agents in FIGS.41A, 41E, and 41F, implying a greater influence of the streaker agent at the bottom rather than the one ontop. In all instances, the swarm agents' show some form of deviation towards the streaker agents in their trajectories.
[0409] FIGS. 42A and 42B show the average orientation of the swarm agents, 0avg, for two instances of initial conditions from FIGS.41A and 41F. Blue and grey lines indicate simulation results with and without the streaker agents, respectively. Average orientation quantifies the changing directionality of the swarm and illustrates the influence of the streaker agents. For FIG. 42A the final swarm direction shifts more than 0.5 radians30°) from —0.16 radians to 0.37 radians when the streaker agents are introduced, clearly marking a significant change in direction towards the desired orientation 0* = n / 5 radians. FIG.42B shows a smaller shift in average orientation, but the influence remains evident.
[0410] FIG. 43 shows the results of 100 simulation trials of different initial conditions for both the cases. The swarm is assumed to be converged if the average swarm orientation reaches within 0-neighborhood of 0* = n / 5 rads during the simulation, where 0=0.5$ rads > 0 is kept constant. Clearly, the streaker agents are successful in guiding the swarm towards a desired direction about 80% of the time, whereas the swarm arbitrarily chooses the right direction only about one fifth of the time when there are no 'informed' agents. Implications of simulation results are discussed in the next section.Discussion
[0411] The simulation results of the bio-inspired feedback control system in presence of high-speed, informed streaker agents suggest that the general orientation of group motion can be influenced significantly by the streaker agents, similar to biological examples in nature. The STMR feedback detects the maximum optic flow value from the surroundings. Since optic flow (Eqn.4.2.1) is a function of the relative speed (and distance) between the observer and the surrounding stimulus, the higher speed of the streaker agents enables the swarm agents to primarily track either of the streaker agents. A swarm is still established without the presence of any streaker agents, as reported in, but previous studies did not establish guidance rules to influence the overall swarm orientation to a desired heading 0*. The baseline case swarm agents thus have different group direction of motion over time, likely based on the initial conditions. The presence of streaker agents is visible in the trajectories of all the mobile agents, resulting in a shift of the swarm orientation in all of the instances. The different swarm orientation for the two cases suggests the higher speed agents generatinggreater optic flow assert greater influence on the swarm and likely influence the swarm orientation for an STMR feedback control rule.
[0412] The plots in FIG.42A and 42B show a similar trend in average group orientation with and without streakers for the first 20 seconds. FIG. 42B even shows that the average orientation crosses over the desired orientation and later comes back up, indicating that the streaker agents' influence increases overtime. The swarm agents are likely influenced by the nearest neighbors at the beginning. Only agents nearest to the streaker agents show initial streaker influence and are the first to align with the streakers. The mobile agents surrounded by other swarm agents may follow the swarm agents in the beginning and establish cohesive group motion among them, seen in the average orientation plots in the first 20 seconds. Only after this cohesion behavior do the mobile agents (blue and grey lines) diverge, indicating the influence of the streaker agents in guiding the direction of the swarm.
[0413] The histogram in FIG.43 shows the convergence results for the two cases considered, simulated over 100 different initial conditions. Convergence is significantly higher in the trials with streaker agents, and establishes the influence of streaker agents in guiding the swarm. It must be noted that the histogram also suggests streaker agents cannot establish desired group orientation 100% of the time, probably owing to factors such as the ratio of streaker agents to swarm agents, relative speeds between the mobile agents, inter-agent distances at t = 0 and the arbitrary gain chosen for the feedback law. Analyzing these parameters could be key to better understand the streaker-swarm interactions and design feedback control systems that guarantee swarm orientation regulation.
[0414] For comparison, both the multi-agent systems are compared in our simulation environment, under identical initial agent distributions, speeds, environmental assumptions, and control parameters, introducing visually salient agents results in coherent adjustments of swarm trajectories while maintaining bounded inter-agent distances with the same initial conditions used in the experiments of FIGS. 44A-44F. Results suggested an increase in convergence to a desired orientation, 0*, when the streaker agents are present. The simulation result in FIG.48A shows one instance of the swarm agents' trajectories for the two feedback laws, STMR feedback from this study and WFI of optic flow based feedback from, for the same initial conditions and streaker agents' trajectories. Only half of the mobile agents using MA-WFI feedback are influenced by the streaker agents and move towards the desired orientation whereas all the mobile agents using STMR feedback change their motion towardsthe streaker agents regardless of their initial angular position. The STMR feedback is not affected by any 'escaped' agents that move in the parallel, opposite direction of the streaker agents, as found in for MA-WFI feedback. FIG. 46B shows the average orientation of all the mobile agents during the simulation. The average orientation of all the mobile agents during the simulation increases towards the desired orientation for both the cases, but the STMR agents reach the desired orientation faster. Average orientation for MA-WFI agents remains close to zero due to the three escaped agents moving in the opposite direction. Setting the desired orientation 0* = 0 helps to eliminate any false positives that may arise from such scenarios. Demonstrated The histogram in FIG. 47 shows the convergence results for 100 trials of different random initial conditions for the two feedback laws being compared. Convergence is clearly higher for agents following the STMR feedback law presented within the present disclosure. These results indicate that implicit information transfer from streaker to swarm agents is more successful for agents using STMR feedback compared to MA-WFI feedback.Summary and conclusions
[0415] This section studies the effects of higher speed, 'informed' agents in guiding the other 'uninformed' agents in two bio-inspired multi-agent feedback control systems: WFI of optic flow-based and STMR, modelled from neurons found in the lobula of insects. No explicit information transfer between the mobile agents is modeled. Previous study has proven orientation stability for a single agent following the MA-WFI feedback in an established formation of mobile agents. In a group of mobile agents using MA-WFI feedback as controller, group orientation alignment is not guaranteed. This study presents simulation results that suggest a group of 'informed' agents might be able to guide the other agents to the desired heading. As for STMR model, the feedback model implemented in this section of the disclosure have been developed earlier in the disclosure (refer to Section III) and suggested average dwell time constraints on the switching control signal for stability. The high-speed streaker agents are expected to provide greater influence on the swarm agents' feedback by producing the maximum optic flow value in the simulation environment, reducing the need to filter limited attention resources. The feedback control system is simulated in Matlab resulting in agent trajectories that demonstrate group direction consensus towards the streaker agents' trajectories. Average orientation plots for all agents with and without the streakers show the influence of streaker agents in changing the swarms' overall motion.Comparing the STMR feedback system with MA-WFI feedback with prior art suggests the swarm agents are more susceptible to the streakers when using the STMR feedback. Future work needs to be done to calculate streaker agents' region of influence on the swarm and to analyze stability of the modified multi-agent system.V. APPLICATION TO UNMANNED AERIAL FLIGHTIntroduction
[0416] This section describes the transition from understanding and developing models of the underlying mechanisms in natural swarms to implementation on multi-agent unmanned fixed-wing aerial systems. Autopilot systems in UAS has in recent years seen improved performance in waypoint tracking and individual flights with the constant development of sophisticated flight controllers such as ArduPilot. These autopilots are not always well equipped for multi-agent swarm platforms. Thus, it is challenging to migrate from ground robot experiments to aerial swarm experiments for implementation of the bio-inspired visually-guided group motion algorithms discussed in the previous sections.
[0417] This section discusses the experimental implementation of STMR feedback control system on a fixed-wing UAV and the experimental results.Methods and ApproachUAV platform
[0418] For initial implementation on fixed-wing aerial vehicles, Commander mPd 1.4m BNF Basic airframe from Horizon Hobby is used, alongwith 'dumm stimulus (PVC pipes) launched from a pneumatic launcher. The motion of the stimulus generate optic flow for the aircraft to track and pursue.Avionics development and integration
[0419] Regarding FIG.48A a controlled aircraft 4800 is fitted with an off-the-shelf 360 degree camera 4805 (FIG. 5.1), i.e., the same camera used for the ground robot experiments described earlier. The camera 4805 connects to a Raspberry Pi 3b onboard computer shown in FIG.48B that takes in the images captured by the camera, runs it through a horizon leveling function and unwraps the 360° image into a linear fit, followed by calculating the instantaneous optic flow 704 for a singular horizontal line. A python script running in the Raspberry Pi calculates the STMR feedback signal that is then sent to the Pixhawk 6c autopilot flight controller 4815, also onboard the aircraft. FIG. 49 shows the STMR feedback loop 4900 for desired heading angle change. Various autopilot controller loops running in the Pixhawkcontroller include yaw control (based on the STMR feedback), roll attitude control, pitch attitude hold and airspeed hold (based on the initial conditions). The STMR feedback control signal 4905 regulates the relative heading angle change only. The rest of the aircraft control is reserved for the on-board Pixhawk autopilot flight controller 4815.STMR feedback loop.TestingSimulation
[0420] Initial testing is done in simulation. Simulation on the loop (SITL) using ArduPilot, MAVProxy and Gazebo software allows implementation and validation of the control system in a simulation environment without risking any expensive hardware. The simulation environment FIG. 49 was set up using a fixed-wing aircraft similar to the aircraft used for actual flight. Python script that use actual flight footage from flight tests for calculation of optic flow and feedback control input can be connected to the SITL environment to simulate aircraft response, and tune parameters.Passive target
[0421] For flight tests, the STMR-controlled aircraft is presented with stimulus from a projectile, launched at a 55° inclination and 180 psi from a launcher 5010, such as a pneumatic launcher. Manual launches are performed to get the stimulus within the field of view of the flying airplane's upward facing camera. The pneumatic launcher 5010 setup is shown in FIG.50B. Flight test includes a planned trajectory 5015 for the flight test is shown in FIG. 50A, and in FIG. 52 data from a flight test shows a projectile trajectory 5020 and the aircraft 4800 for several runs.Active target
[0422] For the final stage of STMR aerial implementation, another aircraft is used as stimulus, i.e., a stimulus aircraft. The stimulus aircraft may be manually flown in a loop at an average airspeed of 25 m / s at 50m altitude as the controlled aircraft 4800 is manually flown in proximity below the stimulus, followed by switching to 'GUIDED' mode to allow STMR response. Repeated runs are performed from different initial conditions of both the aircraft.Results and Discussion
[0423] FIG. 51 shows the output from the camera sensor and its corresponding optic flow calculation from one flight test. The STMR framework considers the highest magnitude of optic flow and regulates orientation of the mobile agent accordingly. Results of the passivetesting are shown in FIG. 52, showing the heading change of the STMR aircraft towards the motion of the projectile. Varying rates of heading change based on the relative initial distance of the STMR aircraft 4800 from the projectile is observed. FIG. 52 shows the results of the active testing for two instances. Similar to the passive testing results, varying rate of heading change can be observed based on the relative distance to the stimulus. This aerial implementation of the STMR feedback framework is the first to use insect visuomotor responses to create GPS-denied, RF communication denied motion coordination in aerial flight with real-time onboard feedback and could provide a pathway for multi-UAV swarm implementations in GPS-denied, RF communication denied environments.
[0424] In FIGS. 53A and 53B illustrated are flight trajectories of the active stimulus aircraft and the STMR aircraft for two instances showing the heading change of the STMR aircraft towards the stimulus aircraft. The markers indicate the final position of the aircraft.
[0425] The next stage of testing in future will be introducing multiple stimulus in the air and recording the response of the controlled aircraft, followed by using the STMR controller on all the aircraft at the same time.Challenges of fixed-wing aerial swarming
[0426] Over the course of the study, numerous flight test attempts had to be made before successful implementation was possible, with various levels of incremental progresses after each flight test. The challenges to aerial fixed-wing group flight arise largely due to the probability of failure from faulty hardware such as servos and broken linkages. If the probability of failure is P for a single airframe, with N airframes it raises to PN, meaning the probability of hardware issues on a flight day rapidly increases in orders of magnitude for multi-airplane experiments. There has been incidents of bad landings leading to structural damage that required days of fixing and buying replacement parts. Instances of bad weather or windy crosswind conditions are often predictable but unavoidable. Sensor calibration and ground testing before flight tests proved vital.
[0427] FIG. 51 shows the measurable optic flow value generated by the stimulus airplane when in view, along with noise coming through the visual signal into the feedback control signal. Motor induced airplane body vibration is a major source of noise coming into the sensor, effects of which is mitigated by implementing a threshold.
[0428] Setting up multiple fixed-wing aircraft in the desired initial condition is not trivial. Our testing was limited to a maximum of 3 airplanes in the air at any given time and manual flightcoordination using the ground control station to get as close to the desired initial condition as possible.Summary and conclusions
[0429] This section of the disclosure discusses the aerial implementation of STMR, a bioinspired multi-agent feedback control framework using optic flow based feedback. Aerial implementation is done on fixed-wing aircraft equipped with a camera sensor for onboard optic flow calculation and real-time feedback. Trajectories from flight tests indicate the controlled aircraft change heading towards the moving stimulus: projectiles and aircraft. Sources of noise to the optic flow measurement are identified and mitigated. Aerial implementation of the STMR feedback framework on fixed-wing aircraft is the first to use insect visuomotor responses to create GPS-denied, RF communication denied motion coordination in aerial flight with real-time onboard feedback and could provide a pathway for large-scale multi-UAV swarm implementations in GPS-denied, RF communication denied environments in future.CONCLUSION AND FUTURE WORK
[0430] This disclosure presents two bio-inspired multi-agent feedback control frameworks based on mathematical models of neural output in the insect visual pathway. The control system presented in Section II uses wide-field integration of optic flow to extract motion cues for navigation. A multi-agent visual model is developed and analyzed for an individual agent operating in a group of other agents. The theoretical model indicates a stable equilibrium trajectory within the neighboring agents. When a minority of mobile agents conducting bioinspired visual feedback are introduced into a planar swarm simulation, results show robustness of mobile agents to perturbations from reference condition. The bio-inspired feedback rule is implemented experimentally using ground microbotsfortwo cases identified by a theoretical observability condition. The experimental performance of the single agent in an established swarm is consistent with the theoretical stability and observability predictions. The results indicate insect visuomotor response can help participate in established swarms, and indicate the importance of optic flow input in maintaining group motion. To address the theoretical observability condition, optic flow enrichment study is presented and implementation of the improved optic flow detection tool in insect lateral-directional flight dynamics simulation is discussed.
[0431] Further, robustness of the WFI-based visual feedback control system is studied. The study considers the system's robustness to sensor noise and configuration deviations, which appear as a nonlinearity in the observation equation often neglected in typical linear feedback. The analysis indicates robust performance can be attained by a static gain Hmcontroller by appropriate choice of gain. Experimental results using an optic flow swarm implementation in robotic systems are used to quantify the actual system performance for static gain controllers. Performance is compared when the system responds to ideal optic flow (no sensor noise) and to camera-based optic flow swarm implementation (sensor noise and configuration uncertainty). The analysis suggests neural gain modulation in insect feedback paths may support robust group motion stabilization.
[0432] The control system presented in Section III uses the maximum optic flow value from the surroundings as feedback for group motion. The study introduces the small-target motion reactive (STMR) swarming approach by designing a concise engineering model of the STMD neurons found in the lobula of numerous insect species. The STMD neuron model identifies the bearing angle at which peak optic flow magnitude occurs, and this angle is used to design an output feedback switched control system. A theoretical stability analysis provides bi-agent stability and state boundedness in group contexts. The approach is simulated and implemented on ground vehicles for validation and behavioral studies. The results indicate despite the lowest connectivity of contemporary approaches (each mobile agent instantaneously regards only a single neighbor), collective group motion can be achieved. The behavioral differences are also highlighted between approaches.
[0433] A swarm guidance mechanism is presented in Section IV by introducing high-speed "streaker" agents in the multi-agent control systems designed earlier. The informed "streaker" agents following a predefined trajectory implicitly transfer information via optic flow to the uninformed swarm agents. Stability analysis for the STMR approach suggest conditional convergence of the group to the desired orientation. Finally, Section V presents the application of the designed system on unmanned aerial vehicles.
[0434] Quite a few clear paths can be determined for future work in this project. Extension of 2D horizontal planar study to 3D optic flow calculation and appending obstacle avoidance techniques to the design will lead to improved aerial swarm implementation for a variety of platforms and environments. A study of the combined MA-WFI and STMR systems can be performed to analyze whether the fusion works to improve or deteriorate performance ofthe overall system. The gain modulation hypothesis suggesting insects modulate their sensory gains for improved robustness could be pursued further in terms of gathering empirical data from biological experiments. Fixed-wing aerial implementation of the designed bio-inspired control frameworks can be further developed to include greater number of mobile agents and compare performance between different frameworks.VI. In-orbit Motion CoordinationINTRODUCTION
[0435] A formation flying mission involves a group of satellite spatially distributed in orbit, capable of inter-agent communication and cooperation to maintain a predetermined formation. This concept has gained significant attention due to its potential to revolutionize future space missions by using low-cost small satellites or CubeSats to enable numerous satellites to function as a distributed network for space missions such as atmospheric sampling, debris detection, assemble large distributed antenna array, enable satellite connectivity, Earth observation and space facilities maintenance. Large distributed networks often run into scalability issues including limited bandwidth, communication delays, vulnerability to faulty hardware, increased likelihood of technical issues and dependence upon the ground station for navigation. On-board sensing and feedback control for autonomous navigation in formation flying enables robustness and scalability, and this is the direction that large scale constellations are taking. As of May 31, 2023, Starlink had completed 25,000 avoidance maneuvers in the last 6 months, averaging 137 avoidance maneuvers per day. These distributed systems are examples of the growth in proliferation, and how scalable growth in orbital systems will benefit from increasing onboard capability, including GPS-free and communication-free ways of completing relative motion tasks.
[0436] Bio-inspired feedback mechanism such as the small-target motion reactive (STMR) controller has been shown to effectively facilitate group motion using computationally inexpensive sensing mechanism and lower bandwidth information transfer without requiring active communication channels [6]. As the LEO orbit gets increasingly crowded with thousands of satellites being launched into mega-constellations, maintaining safe operations and avoiding collisions becomes significant. Fully autonomous approach using on-board sensing provide an efficient and robust form of space navigation with minimal human interference or control. This study considers the STMR feedback controller, and implements in a space satellite formation flying model.
[0437] The concept of formation flying arose from the need for distributed satellite systems capable of performing tasks that are impractical or inefficient for a single large spacecraft. Early work in this area focused on theoretical frameworks for relative motion dynamics in near-Earth orbits, including foundational formulations for describing and predicting inter-satellite motion. These relative motion models enabled the development of missions in which multiple spacecraft operate in coordinated configurations to achieve shared sensing, mapping, or observation objectives.
[0438] Several space missions have demonstrated the feasibility and utility of formation flying. For example, satellite pairs have been employed to cooperatively map planetary fields, observe terrain with high spatial resolution, and maintain precise relative positioning over extended periods. More recently, a growing number of multi-satellite missions have been deployed, including both tightly coordinated formation-flying systems and large distributed constellations. As the number and density of such systems increase, there is a growing interest in autonomous navigation and coordination strategies that reduce reliance on external infrastructure.
[0439] This disclosure considers multi-satellite formation flying in space environments where global positioning signals or continuous inter-satellite communications may be unavailable or unreliable. In such settings, onboard sensing and feedback become important for maintaining relative motion and achieving coordination objectives.
[0440] Biological systems, including flying insects, demonstrate efficient motion control capabilities using simple sensory cues. In particular, optic flow— the apparent motion of visual features across a sensor field— has been shown to support a wide range of navigation and control behaviors. Optic-flow-based feedback has been applied in engineered systems for tasks such as autonomous flight, obstacle avoidance, navigation, altitude regulation, and coordinated group motion. Because optic flow can be extracted using lightweight optical sensors and relatively low computational resources, it is well suited for onboard sensing in space-based platforms.
[0441] In a space formation-flying context, onboard optical sensors may detect optic flow generated by the relative motion of neighboring spacecraft. These measurements can be used to derive control inputs that adjust the motion of a satellite relative to its neighbors, enabling coordinated maneuvers such as rendezvous or formation maintenance. Even short observation windows may be sufficient to extract useful motion information. This disclosureleverages a bio-inspired, optic-flow-based feedback approach to enable formation flying in space using onboard sensing alone
[0442] Assume an Earth-satellite-target scenario. Once the deputy target has been detected and tracked via an optical system, the next stage is chief satellite RPO with the target. A feedback control system for the chief satellite is designed to enable rendezvous with the target. First, the equations of motion of the satellite and the target are considered. In the Earth-centered inertial (ECI) frame, the motion of any object in LEO is affected only by the gravitational force of the Earth and can be expressed as(6.1.1)where Ftis the total force acting on the object i (denotes either the satellite or the target), Meand 7n£are the masses of the Earth and the object respectively, ft]' is the object's acceleration in the inertial frame, G is the universal gravitational constant, 7 is the object position with respect to the Earth. For our desired Earth-satellite-target system, the equations of motion of the satellite S and the target T, based on Eqn. 6.1.1, can be expressed asft] (6.1.2)Twhere u = [axayaz] is the applied control acceleration. For relative motion, the spatial separation between the satellite and target is defined as p = rT— rs.
[0443] For relative motion, the spatial separation between the satellite and target is defined as p = rT - rS. Taking double derivative and substituting from Eqn. 2 yieldsp = fT- fsrrrsP -p— + p— — urs
[0444] Generally, the equations of motion of the satellite is written in a satellite body-fixed rotating frame, with the origin being at the satellite's center of gravity 0s. Consider the framedefined as S = {0s, i,], k where the i axis is oriented along the radius vector rs. k points in the direction normal to the Earth-satellite orbital plane and j completes the right-handed coordinate set. The S frame unit vectors are expressed as follows
[0445] The relative position vector in the S frame is defined asp = xi + y] +zk. (6.1.7)
[0446] Since the S frame rotates with respect to the inertial ECI frame with an angular velocity coB / 7 =the transport theorem can be applied to the inertial derivatives of the relative position vector p.
[0447] This includes (2 co x p) which is the Coriolis acceleration term and (co x (co x p) which is the Centripetal acceleration term. Rearranging Eqn 6.1.8 and substituting Eqn 6.1.3 yield(6.1.9)
[0448] Note, the angular velocity coB / ;denotes the angular velocity of the frame with respect to the inertial ECI frame, which is equal to the rotation rate of the satellite around the Earth such that coB / ;= [00 co]7" = cofc. A skew-symmetric matrix SB / ;can then be defined to replace the cross-productsuch that x p) = SB / / p. Eqn 6.1.9 can be written as
[0449] Carrying out the matrix multiplication and simplifying Eqn. 6.1.11, the component form non-linear equations of relative motion can then be expressed asWhererr3= ||fe + p)|| = [fe + x)2+ y2+ z2]3 / 2(6.1.13)
[0450] In state-space model framework, the non-linear relative equations can be written in the form X = / (X) + Bu such thatwhereX = [x y z x y z]T= [%i x2x3x4x5x6]r(6.1.15) The non-linear dynamics in Eqn. 12 can be linearized by the following assumptions:• Chief satellite orbit around the Earth is circular. Circular orbit means the angular velocity a) is constant and d) = 0.• Spatial separation between the two satellites ( ) is much smaller than the chief satellite's distance from Earth's origin ( rcfrom FIG. 1), meaning |p| «|rc|.The linearized equations can be written asSTMD Neuronal Model
[0451] Literature suggests the STMD neurons found in insects such as dragonflies and hoverflies are highly responsive to a target's size, velocity, contrast, relative distance and direction of motion. The STMD neurons also respond to a single target at any particular time. STMD neurons process the optic flow patterns to extract relevant information about the surroundings. The STMD-model identifies the maximum magnitude of optic flow signal and its corresponding angular position from the entire visual field of view aswhere Q is the optic flow signal defined in a continuous, body-frame-relative spherical coordinate system as developed in Humbert, and y and ft are the respective azimuth and elevation angles (FIG.4,). This STMD-model utilizes the target's size, velocity, relative distance and direction of motion based on the target's optic flow signal.
[0452] For convenience, hereon let's refer to the STMD outputs for the deputy satellite based on measurements from the chief satellite as
[0453] The optic flow signal Q in a spatially sparse environment is defined as a discrete function of the relative motion between the observer and the target objectwhere a> and v represent the observer's angular velocity and relative velocity with respect to the target respectively. The vector r points to the object from the observer along the unit vector Q = r / r. The optic flow signal Q is defined as the time derivative of the unit vector Q and consists of a rotational componentand a translational component Qv.
[0454] The translational component of optic flow Qvcan be described as the relative velocity between the observer and the target v with motion along the radial axis { < v, Q > Q ) removed, then scaled by the inverse of the relative distance r. The translational optic flow can be expressed in the spherical axis (FIG.4) as(6.1.19)
[0455] The rotational compons from the observer's rotational motion. Since the chief's states are known, optic flow generated by the chief's own motion can be calculated and subtracted from the measured optic flow. Then, using Eqn. 19, Eqn. 18 can be expressed asfl 11 RQ — Qa>=Qv=- - L (6.1.20)Lr rJ L J
[0456] The translational optic flow components along the ey-and e^-spherical axes (FIG.4, (left)) can be written assince the [ey,er, e^] frame coincides with the [ex,ey, ez] body relative frame defined in FIG.1, given the optic flow sensor on the chief satellite is positioned to point out along the eyaxis.
[0457] Assuming the relative distance between the two satellites (|p|) is known, the relative position of the deputy satellite can be extracted from the ystmd;2 STMD output as followsand the relative motion can be extracted from the ystmdii STMD output as followsFeedback controller
[0458] Using the states estimated from the STMD output, a control feedback system is designed. The feedback input is defined aswhere K is the proportional gain. The closed-loop system in Eqn. 6.1.16, with the feedback controller in Eqn. 6.1.25, can be expressed asStability analysis of the closed-loop system
[0459] The characteristic equation of the system is given by determinant of si — i4ci) where Aciis the closed-loop system in Eqn.26. The characteristic equation of the closed-loop system(s2+ K) K2s2+ 2K2S + K2+ 2KS3+ 2Ks2+ s4) (6.1.27)
[0460] And the resulting eigenvalues of the closed-loop system are calculated by solving det(s / — i4ci) = 0, which results in
[0461] It can be clearly seen that for any choice of gain K > 0, the real component of the eigenvalues are equal or greater than zero, and thus the system is marginally stable.RESULTS AND DISCUSSION
[0462] The system in Eqns. 6.1.3 and 6.1.26 may be implemented in MATLAB simulation environment for ECI frame and rotating body-fixed S frame respectively. The equations of motion are numerically integrated using MATLAB's built-in ode 45 function, with relative toletance of 1 x 10-12and absolute tolerance of 1 x IO-20. The ISS orbit is chosen as the chief's initial orbit. Example scenarios are run with the target at initial condition given in Table 1.
[0463] Simulation results for the system in the inertial reference frame are plotted in FIGS.55A-58B. FIGS. 59A-61B shows the simulation results for the system in the satellite body-fixed rotating reference frame.Table 1 Target initial conditions for simulation.Summary and Conclusion
[0464] This study derives the non-linear relative equations of motion applied to a chief satellite and a deputy target in the LEO orbit and designs a feedback control system to facilitate multi-satellite RPO. In this space-based embodiment, the computation model 218a calculates the relative state vector in the rotating Hill's reference frame, and the satellite's control logic 218b processes the optical bearing angles to generate thruster actuator commands to maintain orbital boundedness. To enable target rendezvous by the chief satellite, navigation capability is analyzed by designing an on-board optical sensing based feedback controller. The feedback control framework performance is analyzed by means of multiple numerical simulations on a realistic rendezvous scenario with a target in LEO orbit. For the simulation environment, ideal conditions are assumed without the effect of J2 perturbations, solar radiations, sensor noise or any other disturbances. Idealized optic flow equation is used in a spatially discrete environment with the target as the only point-mass object generating optic flow. Results indicate optic flow generated by the target can be used for continuous trajectory correction, and eventually rendezvous. Implementation of the system is performed in both ECI reference frame and the chief's body-fixed rotating Hill-Clohessy-Wiltshire (HCW) frame. Initial results suggest stability and convergence of the relative dynamical system.
[0465] From the above description, it is clear that the inventive concepts disclosed herein are well adapted to carry out the objects and to attain the advantages mentioned herein as well as those inherent in the inventive concepts disclosed herein. While presently preferred embodiments of the inventive concepts disclosed herein have been described for purposes of this disclosure, it will be understood that numerous changes may be made which will readily suggest themselves to those skilled in the art and which are accomplished within the scope and coverage of the inventive concepts disclosed and claimed herein.
Claims
What is claimed is:
1. A system, comprising:a plurality of mobile agents wherein each mobile agent comprises:a sensory system configured to perceive visual information from at least one neighboring mobile agent during a time period;a controller, comprising:at least one processor;a computational model of insect retina-to-motor neuron dynamics coupled to the at least one processor, andcontrol logic, derived from a theoretical multi-agent behavior analysis, that causes the at least one processor to analyze, at instants of time within the time period, said visual information according to said computational model to generate actuator commands; and at least one actuator executing the actuator commands to achieve and maintain a local formation or coordinated motion with at least one neighboring agent.
2. The system of claim 1, wherein the theoretical multi-agent behavior analysis includes a convergence boundedness analysis.
3. A system, comprising:a plurality of mobile agents;a sensory system on each mobile agent configured to perceive visual information from at least one neighboring mobile agent during a period of time; a controller on each mobile agent, comprising:at least one processor, andcontrol logic configured to, at instants of time within the time period, cause the at least one processor to detect, motion blur within the visualinformation caused by the at least one neighboring mobile agent, calculate a bearing angle at which a specific feature in the motion blur occurs within the visual information and generate actuator commands based on the bearing angle; andat least one actuator on each mobile agent executing the actuator commands to achieve and maintain a local formation or coordinated motion with at least one neighboring agent.
4. The system of claim 3, wherein the specific feature in the motion blur is one or more of a peak motion blur magnitude, a peak motion blur minimum magnitude, a blur inflection point, or blur flat points.
5. A method for controlling a plurality of mobile agents to achieve local formations, comprising:providing each of the plurality of mobile agents with a sensory system capable of perceiving visual information from a neighboring mobile agent during a period of time;processing at instants of time within the period of time, by a controller having at least one processor on each mobile agent, said visual information according to a computational model of insect retina-to-motor neuron dynamics to generate actuator commands; andexecuting, by at least one actuator on each mobile agent, said actuator commands to achieve and maintain a local formation or coordinated motion with at least one neighboring mobile agent.
6. The method of claim 5, wherein processing, by the controller having at least one processor on each mobile agent is defined further as detecting motion blur or optic flow within the visual information caused by the at least one neighboring mobile agent, calculating a bearing angle at which a peak motion blur magnitude occurs within the visual information and generating actuator commands based on the bearing angle.
7. A method for controlling a plurality of mobile agents to achieve local formations, comprising:representing visual interactions between mobile agents as a graph-theoretic representation;using said graph-theoretic representation to determine information flow and computational workload among the mobile agent; andcontrolling movement of each mobile agent based on said determined information flow and computational workload, according to a control law derived from a theoretical convergence analysis of said graph-theoretic representation, to achieve and maintain a local formation of the mobile agents.
8. A mobile agent, comprising:an optic flow sensor, configured to generate optic flow signals from visual input; a processor, communicatively coupled to the optic flow sensor, and configured to: identify, from the optic flow signals, a maximum optic flow magnitude and a corresponding bearing angle to a neighboring agent generating said maximum optic flow magnitude, wherein the processor selects said neighboring agent as a target for feedback control; andgenerate a control input based on the bearing angle to the selected target neighboring agent; andan actuator configured to receive the control input and adjust an orientation of the mobile agent relative to the neighboring agent.
9. The mobile agent of claim 8, wherein the processor is configured to implement a switched feedback system wherein the selected neighboring agent may switch over time as different neighboring agents generate the maximum optic flow magnitude, and wherein the processor enforces a dwell time constraint limiting a rate at which switching occurs to ensure boundedness of mobile agent states.
10. The mobile agent of claim 9, wherein the control input implements a pure pursuit guidance law wherein the processor regulates the mobile agent orientation to reduce the bearing angle to the selected neighboring agent.
11. The mobile agent of claim 9, wherein the control input implements a motion camouflage guidance law wherein the processor regulates optic flow generated by the optic flow sensor with respect to the selected neighboring agent toward zero.
12. The mobile agent of claim 9, wherein the dwell time constraint is an average dwell time constraint defined by a minimum average time between switches, such that a number of switches over any time interval is bounded.
13. A mobile agent comprising:a sensory system configured to perceive visual information from at least one neighboring mobile agent during a time period;at least one processor;a non-transitory computer readable medium communicatively coupled to the at least one processor, the non-transitory computer readable medium storing:a computational model of insect retina-to-motor neuron dynamics coupled to the at least one processor, andcontrol logic, derived from a theoretical multi-agent behavior analysis, that causes the at least one processor to analyze, at instants of time within the time period, said visual information according to said computational model to generate actuator commands; andat least one actuator executing the actuator commands to achieve and maintain a local formation or coordinated motion with at least one neighboring agent.
14. The mobile agent of claim 13, wherein the theoretical multi-agent behavior analysis includes a convergence boundedness analysis.
15. A mobile agent, comprising:a sensory system configured to perceive visual information from at least one neighboring mobile agent during a period of time;at least one processor, andcontrol logic configured to, at instants of time within the time period, cause the at least one processor to detect motion blur within the visual information caused by the at least one neighboring mobile agent, calculate a bearing angle at which a specific feature in the motion blur occurs within the visual information and generate actuator commands based on the bearing angle; and at least one actuator on each mobile agent executing the actuator commands to achieve and maintain a local formation or coordinated motion with at least one neighboring agent.
16. The mobile agent of claim 15, wherein the specific feature in the motion blur is one or more of a peak motion blur magnitude, a peak motion blur minimum magnitude, a blur inflection point, or blur flat points.
