Autonomous operations for delivery vehicles and systems
The aerial vehicle system addresses delivery challenges by using sensor data and machine learned models to navigate around obstacles, ensuring reliable and safe delivery of payloads to precise locations.
Patent Information
- Application Number
- PCT/US2025/038646
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-03-06
- Filing Date
- 2025-07-22
- Publication Date
- 2026-11-19
AI Technical Summary
Aerial vehicles face challenges in reliably and safely delivering payloads to precise locations due to unpredictable obstacles, limiting the types of goods that can be shipped and the delivery experience for recipients.
An aerial vehicle system navigates to a delivery area, analyzes data from sensors to identify a payload location, and deploys a secondary vehicle to support the payload, using machine learned models to predict obstacles and determine optimal delivery pathways.
Enables reliable and safe delivery of diverse goods to precise locations by navigating around obstacles, enhancing the delivery experience and reducing noise and safety risks for recipients.
Smart Images

Figure US2025038646_19112026_PF_FP_ABST
Abstract
Description
Docket Number P312830.W0.01AUTONOMOUS OPERATIONS FOR DELIVERY VEHICLES AND SYSTEMSCROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to U.S. Provisional Patent Application No.63 / 673,991, filed on July 22, 2024 and titled “Autonomous Operations for Delivery Vehicles and Systems,” and U.S. Provisional Patent application No. 63 / 767,728, filed March 6, 2025 and titled “Autonomous Operations for Delivery Vehicles and Systems,” each of which is incorporated herein in the entirety and for all purposes.FIELD
[0001] The described embodiments relate generally to an aerial vehicle, such as one that may be used to deliver payloads (e.g., packages), as well as other types of autonomously operated components or vehicles.BACKGROUND
[0002] Aerial vehicles, such as airplanes and unmanned vehicles (e.g.. drones), have many uses. Recently, aerial vehicles are becoming a viable option for package delivery vehicles. Such aerial vehicles can take many forms, such as, but not limited to, rotorcraft (e.g., helicopters, quadrotors, and so on) as well as fixed-wing aircraft. During delivery, all or portion of the flight commands to the aerial vehicle may be autonomously generated or executed. However, the unpredictability of obstacles encountered during flight or at pickup or delivery locations for a package, whether in the air or on the ground at pickup or delivery' locations may limit the types of goods that may be shipped by conventional aerial vehicles.
[0003] As aerial vehicles are used more frequently for package deliveries, there is a need for improved overall systems that allow reliable and at least partially autonomous delivery of a package to a delivery location. Further, there is a need for a system that provide an ability to safely carry' a diverse array of goods in a wide range of-1- 4910-9932-4237\3Docket Number P312830.W0.01 environments to precise locations using autonomous delivery systems that create a smooth and pleasant delivery experience for a recipient.SUMMARY
[0004] In one example, a method of navigating an aerial vehicle system for a delivery7of a pay load is disclosed. The method includes navigating a primary aerial vehicle of the aerial vehicle system to a delivery area, analyzing data including information from one or more sensors in communication with the aerial vehicle system to identify a payload location within the delivery area based on a landability assessment, and deploying the pay load to the pay load location.
[0005] In another example, the pay load location is a region of the delivery area defined by one or more of objects or terrain sufficient to land an aerial vehicle of the aerial vehicle system or place a payload.
[0006] In another example deploying includes navigating a secondary7aerial vehicle of the aerial vehicle system dependently coupled with the primary aerial vehicle to the payload location, and the secondary aerial vehicle supports the payload.
[0007] In another example, the method further includes analyzing refinement information corresponding to the delivery7area, the data including the refinement information.
[0008] In another example, the refinement information includes data corresponding to a user input and the user input identifies the delivery area.
[0009] In another example, the user input identifies a preferred payload location or regions to avoid at the delivery7area.
[0010] In another example, the refinement information includes the data from the one or more sensors of the aerial vehicle system, and the data includes information corresponding to one or more objects or characteristics of the delivery area.
[0011] In another example, the method further includes determining an actual position of the primary aerial vehicle relative to the delivery area using the refinement information, navigating the primary aerial vehicle from the actual position to the delivery area.
[0012] In another example, the method further includes identifying the actual position by capturing the refinement information by the one or more sensors and comparing the-2- 4910-9932-4237\3Docket Number P312830.W0.01 captured refinement information with preexisting information associated with the delivery area.
[0013] In another example, the method further includes determining a secondary vehicle flight path from the primary aerial vehicle to the payload location using the refinement information and the refinement information includes one or more predetermined flight paths associated with the delivery area.
[0014] In another example, the method further includes capturing images of the delivery area by the one or more sensors, and the data includes the images, analyzing the images by a machine learned model to determine one or more delivery’ pathways to the payload location, and the delivery pathways are based at least in part on a presence or position of one or more of objects, features, or characteristics of the delivery area.
[0015] In another example, the delivery pathway is a route from the primary aerial vehicle to the payload location without contacting one or more obstacles based on the presence or position of the one or more objects, features, or characteristics of the delivery area.
[0016] In another example, the machine learned model to identifies a distance corresponding to a portion of the delivery’ area by analyzing the data.
[0017] In another example, the machine learned model identifies the one or more objects, features, or characteristics of the delivery area by analyzing the data.
[0018] In another example, the machine learned model includes a first machine learned model identifying the one or more objects, features, or characteristics of the delivery area, and a second machine learned model determining a distance corresponding to the one or more objects, features, or characteristics of the delivery area.
[0019] In another example, the method further includes delivering the payload at the payload location, and the secondary aerial vehicle lands on or is positioned above the payload location and releases the payload.
[0020] In another example, the one or more sensors are separately located from the primary aerial vehicle or the information from the one or more sensors is captured prior to navigation of the primary aerial vehicle.
[0021] In another example, the data is provided to the aerial vehicle system prior to navigation of the primary aerial vehicle.-3- 4910-9932-4237\3Docket Number P312830.W0.01
[0022] In another example, the data is communicatively received by the aerial vehicle system during navigation of the primary aerial vehicle.
[0023] In another example, the method includes capturing images of the delivery area, and analyzing the captured images by a machine learned model to identify the payload location based at least in part on distance information and a type of object at the delivery' area obtained from the captured images.
[0024] In another example, the machine learned model identifies the payload location based on a pattern between the type of object and the distance information.
[0025] In one example, a method of determining a payload location for deliver}' of a payload by an aerial vehicle system is disclosed. The method includes navigating a primary aerial vehicle of the aerial vehicle system to a delivery location responsive to a delivery request, identifying one or more objects at the delivery location, identifying a payload location at the delivery location, and the payload location is a portion of the delivery location meeting one or more landability criteria, determining a delivery' pathway from the primary aerial vehicle to the payload location, and a payload package from the primary aerial vehicle along the delivery’ pathway.
[0026] In another example, navigating comprises utilizing a secondary aerial vehicle, and the payload package is releasably coupled to the secondary' aerial vehicle. In another example, the method further includes after navigating the primary aerial vehicle to the delivery location, rotating the primary’ aerial vehicle or the secondary’ aerial vehicle about an axis to search for one or more aerial objects at the delivery location.
[0027] In another example, the axis is a vertical axis, and the primary aerial vehicle rotates about the vertical axis while the secondary' aerial vehicle is deployed from the primary aerial vehicle.
[0028] In another example, the method further includes monitoring a position of the secondary aerial vehicle relative to the primary aerial vehicle during navigation of the delivery’ pathway.
[0029] In another example, the primary aerial vehicle includes one or more sensors, and the one or more sensors capture images of the secondary aerial vehicle to determine the position of the secondary aerial vehicle relative to the primary aerial vehicle.
[0030] In another example, the method includes capturing refinement information by one or more sensors of the aerial vehicle system.-4- 4910-9932-4237\3Docket Number P312830.W0.01
[0031] In another example, spaces occupied by one or more objects are identified by a first machine learned model by analyzing refinement information captured by one or more sensors of the aerial vehicle system.
[0032] In another example, a second machine learned models determines a distance corresponding to the objects by analyzing the refinement information captured by the one or more sensors.
[0033] In another example, the first machine learned model or the second machine learned model are trained by analyzing training data, and the training data includes one or more annotations identifying portions of the training data representative of environmental objects similar to the objects or distances to the environmental objects.
[0034] In another example, navigating the secondary aerial vehicle includes determining the distances to the objects and selecting a pathway through an at least partially vertical space between the primary aerial vehicle and the payload location.
[0035] In another example, the training data includes one or more of captured images or computer generated images.
[0036] In another example, the method further includes delivering the payload at the payload location, and the secondary aerial vehicle lands on or is positioned above a feature at the payload location and releases the payload.
[0037] In another example, the method includes capturing images of the delivery location, analyzing the captured images by a machine learned model to identify the one or more objects and associate the objects with distance information, and identifying the payload location by the machine learned model based on a pattern between the objects and the distance information.
[0038] In another example, the machine learned model is trained to identify the payload location based on an association between the pattern and the one or more landability criteria.
[0039] In one example, a method of navigating an aerial vehicle is disclosed. The method includes determining a first position of the aerial vehicle, determining the first position is in proximity to a known location by detecting a fiducial at the known location by one or more sensors of the aerial vehicle, and the fiducial is associated with a predetermined flight path to a second position, generating a flight path from the first-5- 4910-9932-4237\3Docket Number P312830.W0.01 position to the second position based on the predetermined flight path, and navigating the aerial vehicle from the first position to the second position.
[0040] In another example, the first position is determined by the one or more sensors.
[0041] In another example, the one or more sensors include a camera to capture the fiducial, and the fiducial is a first fiducial corresponding to a first portion of the predetermined flight path and a second fiducial corresponds to a second portion of the predetermined flight path.
[0042] In another example, the fiducial is in communication with the aerial vehicle by one or more of visible indicators, electromagnetic emissions, or acoustic emissions.
[0043] In another example, during navigation of the generated flight path the aerial vehicle limits or reduces the capture of or capturing information by the one or more sensors.
[0044] In another example, navigating from the first position to the second position includes deploying a secondary aerial vehicle from a primary aerial vehicle at the first position, and the aerial vehicle is the secondary aerial vehicle, and navigating the secondary aerial vehicle from the primary aerial vehicle to the second position.
[0045] In another example, the first position is a docking location and the second position is a flight corridor.
[0046] In another example, the first position is a flight corridor and the second position is a payload delivery location or a docking location.
[0047] In one example of a method of navigating an aerial vehicle of an aerial vehicle system, the method includes determining a first position of the aerial vehicle, identifying a known location by one or more sensors of the aerial vehicle system, wherein the known location is associated with one or more objects with known positions at the known location, determining the first position is in proximity to or at the known location, generating a flight path from the first position to the second position based at least in part on one or more predetermined flight paths associated with the known location to avoid the one or more objects, and navigating the aerial vehicle from the first position to the second position.
[0048] In another example, during navigation of the generated flight path the aerial vehicle limits or reduces the capture of information by the one or more sensors.-6- 4910-9932-4237\3Docket Number P312830.W0.01
[0049] In another example, navigating from the first position to the second position includes deploying a secondary aerial vehicle from a primary aerial vehicle at the first position, wherein the aerial vehicle is the secondary aerial vehicle, and navigating the secondary aerial vehicle from the primary aerial vehicle to the second position.
[0050] In another example, the first position is a docking location and the second position is a flight corridor.
[0051] In another example, the first position is a flight corridor and the second position is a payload delivery location or a docking location.
[0052] In one example, a method of delivering a payload is disclosed. The method includes navigating a primary unmanned aerial vehicle to a delivery area, determining a delivery pathway from the primary unmanned aerial vehicle to a payload location within the delivery area, and navigating a secondary unmanned aerial vehicle from the primary unmanned aerial vehicle downwards within the delivery pathway to the payload location.
[0053] In another example, the secondary unmanned aerial vehicle is a dependent vehicle to the primary unmanned aerial vehicle.
[0054] In another example, the method further includes capturing refinement information corresponding to the delivery area by sensors coupled to the primary unmanned aerial vehicle or secondary' unmanned aerial vehicle, and identifying, by at least a first machine learned model while the primary unmanned aerial vehicle is at the delivery area, one or more environmental features at the delivery area or distances corresponding to the one or more environmental features.
[0055] In another example, the first machine learned model identifies the one or more features at the delivery' area from data generated from the refinement information corresponding to the one or more environmental features, and a second machine learned model identifies the distances associated with the data from the refinement information corresponding to the one or more environmental features.
[0056] In another example, the delivery' pathway is determined, at least in part, by identifying a pattern between the presence of and distance to environmental obstacles at the delivery area to identify the payload location, and determining a route between the primary unmanned aerial vehicle and the pay load location avoiding contact with the identified environmental obstacles.-7- 4910-9932-4237\3Docket Number P312830.W0.01
[0057] In another example, the payload location is identified based, at least in part, on the presence of environmental objects between the primary unmanned aerial vehicle and a portion of the delivery area.
[0058] In another example, the payload location is determined by the presence of or distance to environmental obstacles at the delivery area having a primary purpose unrelated to the aerial vehicle system.
[0059] In another example, the primary unmanned aerial vehicle navigates to the delivery area by information captured by one or more sensors during navigation.
[0060] In another example, the primary unmanned aerial vehicle navigates to the delivery area utilizing information generated prior to navigation.
[0061] In another example, the primary unmanned aerial vehicle navigates to the delivery area by one or more of information communicated to the primary unmanned aerial vehicle during navigation, information captured by the one or more sensors during navigation, or information generated prior to navigation.
[0062] In one example, a method of training a machine learned model to analyze an environment is disclosed. The method includes generating training data including information representative of the environment, and at least a portion of the training data is computer generated, assigning one or more annotations to the training data identifying one or more portions of the training data representative of at least one characteristic of the environment, training the model to identify the at least one representative characteristic by analyzing the training data, and deploying a trained model in operative communication with an aerial vehicle system to identify an environmental characteristic of a real world environment corresponding to the at least one characteristic.
[0063] In another example, at least a second portion of the training data includes information captured by one or more sensors representative of the real world environment.
[0064] In another example, the at least one characteristic is a classification of a feature of an environment, and the annotation is an identifier to assist in classifying the feature.
[0065] In another example, the trained model identifies one or more environmental objects positioned in the real world environment based on captured information corresponding to the at least one characteristic.-8- 4910-9932-4237\3Docket Number P312830.W0.01
[0066] In another example, the at least one characteristic is a depth or distance to a feature of an environment, and the annotation is a true depth or distance corresponding to one or more portions of the training data including the feature.
[0067] In another example, the trained model identifies depths or distances to one or more environmental objects in the real world environment based on captured information corresponding to the at least one characteristic.
[0068] In another example, inputs for training the trained model include refinement information captured by one or more sensors of an unmanned aerial vehicle.
[0069] In another example, the training data includes information corresponding to an output from a second trained model, and the annotation is the identification of at least a portion of the output relevant to determining the characteristic.
[0070] In another example, the at least one characteristic of the environment includes an object classification and distance information to the object, and the model is trained to identify a location in the real world environment based at least on patterns between the object classification and the distance information to patterns in the training data between similar features.
[0071] In another example, after deployment of the trained model, the trained model analyzes captured images data of the real world environment to identify a payload location based at least in part on similarities to the patterns between the object classification and the distance information.
[0072] In one example, a method of navigating an aerial vehicle is disclosed. The method includes capturing information of an environment by one or more sensors, identifying a passive identifier corresponding to information about the environment, generating a flight path from a first position to a second position based on the information about the environment, and navigating the aerial vehicle from the first position to the second position.
[0073] In another example, the passive identifier is associated with a docking structure, and the information about the environment includes a location of the docking structure.
[0074] In another example, the docking structure is one of a plurality of docking structures, and the passive identifier identifies the docking structure from the plurality of docking structures.-9- 4910-9932-4237\3Docket Number P312830.W0.01
[0075] In another example, the method further includes analyzing the captured information from the one or more sensors by one or more machine learned models to identify the passive identifier.
[0076] In another example, the machine learned models are stored at a memory component onboard the aerial vehicle.
[0077] In another example, navigating the primary aerial vehicle of the aerial vehicle system to a delivery location includes rotating the primary rotating the primary aerial vehicle or the secondary aerial vehicle about an axis during takeoff to search for one or more aerial objects during ascension.
[0078] In another example, navigating the primary aerial vehicle of the aerial vehicle system to a delivery location includes rotating the primary rotating the primary’ aerial vehicle or the secondary aerial vehicle about an axis during docking to search for one or more aerial objects during descension.
[0079] In another example, navigating the primary aerial vehicle of the aerial vehicle system to a delivery location includes rotating the primary rotating the primary’ aerial vehicle or the secondary aerial vehicle about an axis during execution of a flight plan to search for one or more aerial objects along the flight plan.
[0080] In another example, the information captured by one or more sensors representative of the real world environment includes image frames, location data, and velocity data captured during a prior flight of an aerial vehicle in the real world environment.
[0081] In another example, generating training data including information representative of the environment includes generating synthetic training data representing an execution of a flight plan by an unmanned aerial vehicle based on the image frames, location data, and velocity data simulating a capture during a prior flight, including: determining a difference in velocity between the simulated prior flight and an expected velocity’ of the unmanned aerial vehicle based on the captured velocity’ data; sampling the captured image frames at a sampling rate based on the difference in velocity; and generating a synthetic representation of the execution of the flight plan by the unmanned aerial vehicle based on the sampled captured images.
[0082] In another example, generating training data including information representative of the environment includes generating synthetic training data of a three- -10- 4910-9932-4237\3Docket Number P312830.W0.01 dimensional representation of the environment based on the image frames, location data, and velocity data captured during a prior flight, including: identifying an object in the real world environment based on the image frames: identifying a location of the object based on the location data: identifying a first image frame captured within a threshold distance of the object based on the location data; and generating a three-dimensional representation of the object based on the first image frame.
[0083] In another example, identify ing the payload location at the delivery location includes: identifying one or more potential locations at the delivery location that meet the one or more landability criteria; generating a ranking of the one or more potential locations based on a ranking preference; and identifying the highest ranked potential location as the payload location.
[0084] In another example, generating the ranking of the one or more potential locations based on the ranking preference includes: receiving a user input comprising a preference for a payload location; updating the ranking preference based on the user input; and ranking the one or more potential locations based on the updated ranking preference.
[0085] In another example, generating the ranking of the one or more potential locations based on the ranking preference includes: identifying a semantic feature of the one or more potential locations, where the semantic feature includes at least one of a geometry of the one or more potential locations, a classification of the one or more potential locations, or a classification of an object identified at the one or more potential locations; ranking the semantic feature of the one or more potential locations based on the ranking preference; and ranking the one or more potential locations based on the ranking of the semantic feature.
[0086] In another example, navigating the payload package from the primary aerial vehicle along the delivery pathway further includes delivering the payload package at the identified payload location; receiving a user input comprising feedback regarding the delivery of the payload package at the identified payload location; and updating the ranking preference based on the user input.
[0087] In one example, a method for delivering a payload by an aerial vehicle system is disclosed. The method includes capturing an image of a delivery’ area by a sensor of an aerial vehicle, analyzing the image by a machine learned model trained to identify a-11- 4910-9932-4237\3Docket Number P312830.W0.01 payload location within the delivery area based on a landability assessment, and deploying a dependent vehicle operatively associated with the aerial vehicle to deliver the payload to the payload location.
[0088] In another example, the payload location is a region of the deliver} area defining a stable area to land the dependent vehicle or release the payload.
[0089] In another example, analysis by the machine learned model includes one or more of determining distance or classifications of objects captured within the image.
[0090] In another example, the machine learned model is trained to predict the payload location by analyzing the images to identify a pattern between objects present at the delivery area and distances to the objects.
[0091] In another example, the method further includes analyzing the images by the machine learned model to determine one or more delivery pathways to the payload location, the delivery pathw ays defined as a volume of airspace navigable by at least the dependent vehicle between the aerial vehicle and the pay load location.
[0092] In one example, a method of delivering a payload is disclosed. The method includes capturing an image of a delivery area by a sensor, analyzing the image by a machine learned model trained to predict a payload location within the delivery area based on patterns between at least spaces occupied by objects and distances to the spaces occupied by the objects within the delivery area, directly predicting the payload location from by the machine learned model, and delivering the payload to the payload location.
[0093] In another example, the payload location is predicted without independently identifying the objects occupying the spaces or the distances to the objects.
[0094] In another example, the image is captured before or during a deliver}' operation for delivery the payload.
[0095] In another example, two images are captured from at least tw o different orientations, and the distance to objects are inferred by comparing differences between the different orientations.
[0096] In another example, the machine learned model is trained by analyzing training data including information corresponding to classifications of objects or distances to objects to predict or identify’ the patterns.-12- 4910-9932-4237\3Docket Number P312830.W0.01
[0097] In another example, the method further includes generating, by the machine learned model, a three dimensional model of the delivery area to predict the pay load location.
[0098] In addition to the exemplar} aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the drawings and by study of the following description.BRIEF DESCRIPTION OF THE DRAWINGS
[0099] The disclosure will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements, and in which:
[0100] Fig. 1 is a rear left isometric view of an aerial vehicle system including a first aerial vehicle and a second aerial vehicle.
[0101] Fig. 2 depicts a front right perspective view of the example aerial vehicle system of Fig. 1 including a deployed secondary aerial vehicle.
[0102] Fig. 3 depicts a schematic diagram of components of the aerial vehicle system.
[0103] Fig. 4 depicts a top right perspective view of an example secondary vehicle delivering an example payload.
[0104] Fig. 5 depicts a cross-section view of the aerial vehicle system of Fig. 1 taken along line 5-5 in Fig. 1.
[0105] Fig. 6 depicts a cross sectional view of a wingtip of the primary aerial vehicle including a sensor assembly taken along lines 6-6 in Fig. 2.
[0106] Fig. 7 depicts a bottom plan view of an example secondary vehicle.
[0107] Fig. 8 is a schematic flow diagram of data between compute resources of aerial vehicle system, in accordance with various examples of the disclosure.
[0108] Fig. 9 illustrates example input to and output from depth estimation and semantic understanding models, in accordance with various examples of the disclosure.
[0109] Fig. 10 illustrates example depth information generated using the semantic understanding model and / or the depth estimation model.
[0110] Fig. 11 illustrates a flow chart for an example method of navigating an aerial vehicle in an environment using depth information for the environment, in accordance with various examples of the disclosure.- 13-4910-9932-4237\3Docket Number P312830.W0.01
[0111] Fig. 12 illustrates a flow chart for an example method of training neural networks for navigation of an aerial vehicle, in accordance with various examples of the disclosure.
[0112] Fig. 13 illustrates a flow chart for an example method of navigating a secondary7aerial vehicle from a primary7aerial vehicle to a delivery7location, in accordance with various examples of the disclosure.
[0113] Fig. 14 illustrates a flow chart of an example method for navigating an aerial vehicle system for a delivery, in accordance with various examples of the disclosure.
[0114] Fig. 15 illustrates a flow chart of an example method for determining a payload location for delivery of a payload by an aerial vehicle system, in accordance with various examples of the disclosure.
[0115] Fig. 16 illustrates a flow chart of an example method for navigating an aerial vehicle system, in accordance with various examples of the disclosure.
[0116] Fig. 17 is a schematic diagram of an example computer system for implementing various embodiments in the examples described herein.DETAILED DESCRIPTION
[0117] The description that follows includes sample sy stems, methods, and apparatuses that embody various elements of the present disclosure. However, it should be understood that the described disclosure may be practiced in a variety of forms in addition to those described herein.
[0118] The examples described herein are related to methods and systems that enable autonomous control of various devices, such as aerial vehicles, terrain vehicles (e.g., automobiles), and other remotely or autonomously operated components. Embodiments disclosed herein help to ensure that the vehicles detect and navigate around obstacles, such as vegetation, wires, structures / buildings, animals, and other vehicles while also steering / directing the vehicles to a desired positioning and orientation to enable completion of a task, such as delivering or receiving a pay load. For example, in some embodiments, the system may detect and navigate an aerial vehicle to a payload delivery location, while navigating around known and unknown or unexpected obstacles.
[0119] In some embodiments, the autonomous systems may be implemented within aerial vehicles or systems utilizing aerial vehicles. The aerial vehicles may be utilized in-14- 4910-9932-4237\3Docket Number P312830.W0.01 a delivery system configured to pick up a payload or a package at a shipping location and deliver a payload or package to a delivery location in a rural and / or urban environment. It should be noted that while various features and components are discussed with respect to an aerial vehicle or aerial vehicle system, these features and components can be used separate from the aerial vehicle and / or in various combinations with each other. As such the discussion of any particular implementation is meant as illustrative only.
[0120] In some embodiments, an aerial vehicle system includes a first or primary aerial vehicle (e.g., a main or carrier aerial vehicle) and a secondary aerial vehicle (e.g., dependent aerial vehicle). In some examples, more than one secondary vehicle may be associated with a primary aerial vehicle. In these instances, the first aerial vehicle may act to support and / or transport the secondary aerial vehicle to a location and the second aerial vehicle can be deployed from the first aerial vehicle, e.g., the second aerial vehicle can be deployed from a first height and descend to a second height or location, such as to deliver a package. The dependent vehicle may be tethered, such as by a cable or other connection, to the first aerial vehicle and may have some range of motion or freedom to move relative thereto. In some embodiments, the first aerial vehicle may be able to remain at an elevated location, such as by hovering, and may reduce the noise, disruption, and safety7risks experienced by humans and animals on the ground or delivery location as the second vehicle may be quieter than the first aerial vehicle. In some examples, the first aerial vehicle can be optimized for longer flight paths and the second aerial vehicle can be optimized for an enhanced delivery experience to a human on the ground or delivery location. The payload may be a good or package including consumer goods, food, medical supplies, or other items. Additional variations of the primary aerial vehicle and / or the secondary aerial vehicle may include types of rotorcraft (e.g., helicopters, quadrotors, and so on) or similar vehicles that generate thrust for movement through air, as well as fixed-wing aerial vehicles.
[0121] The locations may include a retail, wholesale, industrial, mail carrier, or other site in which payloads and packages are processed for delivery to a customer. The delivery location may include a payload location designated as a specific portion of a building or area, such as a door, a window, a deck, a roof, a parking area, or other locations accessible by a delivery7recipient.-15- 4910-9932-4237\3Docket Number P312830.W0.01
[0122] The first aerial vehicle may include flight assemblies enabling different types of flight. For example, the first aerial vehicle may include fixed wings and a cruise propeller configured for forward or cruise flight motion. The first aerial vehicle may include one or more propeller assemblies configured for hover or similar motion. The cruise propeller may be articulable (e.g., rotatable) between a cruise flight and hover flight position.
[0123] In many embodiments, the second aerial vehicle may be coupled to the first aerial vehicle, e.g., by a tether, cable, or the like. In some embodiments, the second aerial vehicle may be stowed within a portion of the first aerial vehicle, such as a cavity or bay, for a first portion of the mission and deployed and retracted for a second portion of the mission.
[0124] The secondary aerial vehicle may have separate drive abilities for the second aerial vehicle to steer itself without or to supplement steering by the first aerial vehicle. For example, the secondary aerial vehicle may include a propulsion assembly for the secondary aerial vehicle to move relative to the primar ’ aerial vehicle. The propulsion assembly may include thrusters to provide active control or enable generation of thrust irrespective of the position of the secondary aerial vehicle.
[0125] The body of the secondary' aerial vehicle may include a volume or feature to house the payload, e.g., a payload bay. In some examples, the payload bay may be accessible through one or more apertures by one or more selectively openable assemblies, such as a lid or doors, to place a payload in the bay or to remove the payload from the bay at the delivery location.
[0126] The aerial vehicles may include one or more sensors that collect data to assist in the operation of the aerial vehicle. The sensors may collect information about a surrounding environment or operations of the aerial vehicles in the environment. For example, the aerial vehicles may include a plurality of sensor assemblies such as audio or sound sensors, visual sensors such as cameras, depth sensors, or the like. In some examples, the sensors may detect or capture information regarding the environment, such as obstacles, weather information, and the like, and may include a computer and / or be in communication with a processor to generate information or decisions regarding flight, docking, landing, and / or other maneuvers of the aerial vehicles.-16- 4910-9932-4237\3Docket Number P312830.W0.01
[0127] The various sensors may be used in combination with processing elements or other systems to navigate the aerial vehicles. For example, the aerial vehicles may be operated remotely or completely autonomously. Processing or computing elements in communication with or integrated into one or more aerial vehicles may analyze the information gathered by the sensors to understand an environment for navigation or delivery by one or more of the aerial vehicles. The processing elements may model the environment, generate flight paths through the environment, or the like.
[0128] The various computing elements or processors in communication with the sensors may be associated with or execute operations of machine learned models, databases, or the like to assist in categorizing or understanding sensed information. For example, a machine learned model may be trained to identify or determine maneuvers based on objects in images or otherwise captured by the sensors within an environment. The various computing elements may identify flight paths or maneuvers within the environment based on the identified objects, or positions of the objects relative to the aerial vehicles. The primary aerial vehicle and secondary aerial vehicle may be in operative communication with each other, such as through wireless networks, cell networks, radio frequencies, wired, or other communication methods.
[0129] The machine learned models may individually identify7objects or distances to portions of the environment, or in some examples model the environment. In some examples, the machine learned models may generate a three dimensional model or routes for navigation by an aerial vehicle without separately classifying objects or distances. For example, the aerial vehicles can include processing elements to execute a trained machine learned model to identify7regions suitable to land or navigate a UAV, or a payload, based on images or other data captured about an environment. By utilizing a single model to identify payload locations, navigation routes, and / or to model an environment, processing demands can be reduced in comparison to utilizing separate models that perform only a portion or discrete aspects of interpreting an environment.
[0130] In some examples, the machine learned models may be trained utilizing data representative of delivery environments or flight paths. The representative, or training, data may be captured by aerial vehicles or sensors similar to the aerial vehicles and / or may be computer generated. Training data gathered by aerial vehicles may provide characteristic features or conditions experienced during operation. For example, captured-17- 4910-9932-4237\3Docket Number P312830.W0.01 training data may include images or sensor readings related to various objects such as power lines, landscaping, roads, or the like as may be encountered in a delivery environment. Computer generated or synthetic training data may be tagged or otherwise annotated with accurate or defined features to generate accurate or verified training data. For example, synthetic training data may be renders of an environment similar to a real world delivery location. The synthetic training data may be generated to include scenarios missing in existing training data or that is difficult to gather. For example, captured training data may not include particular types of objects and synthetic training data may be generated to include such objects, such that the trained machine learning models are able to identify such objects.
[0131] The training data may be annotated by a user or a computing system to include various identifiers of the information included in the training data, such as types of objects in captured images, movements associated with or characteristics to assist in identifying the objects, or distances between or to the obstacles. Such annotated data may be utilized to train the model using supervised learning methods. In some examples, machine learned models may be trained using unsupervised or other training methods. Trained machine learned models may analyze or predict the presence of various features, objects, distances, or flight paths through a captured environment and compare the predictions to the various identifiers. The comparisons may be used to refine and update the models.
[0132] In examples of the aerial vehicle system utilizing a machine learned models, such as world modeling, route planning, or identifying payload locations, later captured information or user (e g. customer or operator) feedback can be included in the training data to improve the model. For example, customer feedback can provide data to validate accurate environmental modeling or indicate errors in modeling. In such examples, it may be beneficial to use single machined learned model for performing these features, such that the processing demands can be reduced by training only a single model, rather than multiple models, with the later received or captured information.
[0133] At operation, the primary aerial vehicle may navigate to a delivery location autonomously using the one or more sensors. For example, a delivery request may be received and identify a payload to be delivered at the delivery location. At the delivery location, the primary aerial vehicle and / or the secondary aerial vehicle may identify a-18- 4910-9932-4237\3Docket Number P312830.W0.01 payload location, which may be a position at the delivery location to deliver the payload. For example, the machined learned models can identify or model the environment to recognize requested delivery locations or identify areas suitable for delivery within the area of the delivery location. For example, the payload location may be determined by the presence or lack thereof of objects or characteristics of the location. In one example, the compute resources or machine learned models may identify' various features or characteristics of the location, such as an object or region satisfying conditions for a suitable payload location, or one or more obstacles between the primary aerial vehicle and a potential payload location. In some examples, captured information can be analyzed by the machine learned model to identify the payload location or routes to the payload location without individually identifying objects or distances to reduce processing demands.
[0134] In some examples, the payload location may be predetermined and recognized by the system. In examples where the location is known, predetermined flight paths may be generated or used by the aerial vehicles to navigate to the payload location. The predetermined flight paths may reduce processing demands or provide maneuvers around known objects. Such predetermined flight paths may also, in some examples, be modified during flight based on detected obstacles, conditions within an environment including the payload location, and the like.
[0135] After arriving at the delivery location, the secondary aerial vehicle may descend from the primary aerial vehicle toward the payload location (e.g., the ground or a landing platform) at the delivery location. In some examples, the payload location may be determined before or after releasing the secondary aerial vehicle from the primary' aerial vehicle. For example, an initial payload location may be determined prior to deploying the secondary aerial vehicle, and the initial payload location may be updated as new information is captured by the sensors.
[0136] Reference will now be made to the accompanying drawings, which assist in illustrating various features of the present disclosure. The following description is presented for purposes of illustration and description. Furthermore, the description is not intended to limit the inventive aspects to the forms disclosed herein. Consequently, variations and modifications commensurate with the following teachings, and skill and knowledge of the relevant art, are within the scope of the present inventive aspects.-19- 4910-9932-4237\3Docket Number P312830.W0.01
[0137] Turning to the figures, Figs. 1 and 2 depict example aerial vehicles 102, 200 of an aerial vehicle delivery system 100 for the aerial delivery of a payload 101 (shown in FIG. 4), such as packages, food, or other items, to a delivery location. The aerial vehicle system 100 may include a first aerial vehicle or primary aerial vehicle 102 and a second or secondary’ aerial vehicle 200. The secondary7aerial vehicle 200 may be releasably coupled or associated with the first aerial vehicle 102. For example, the secondary aerial vehicle 200 may be a dependent aerial vehicle. A dependent aerial vehicle may be controlled by, in communication with, tethered to, or otherwise dependent on another aerial vehicle (e.g., the primary aerial vehicle 102). The first aerial vehicle 102 may include a vehicle compartment 128 defined within a fuselage 120 to receive all or a portion of the second aerial vehicle 200. In some embodiments, the second aerial vehicle 200 may nest or be positioned within the first aerial vehicle 102 in at least one configuration. The second aerial vehicle 200 may form a portion of an outer surface of the fuselage 120 (e.g., a bottom 208 of the second aerial vehicle 200 may be exposed to the environment during flight). In other embodiments, the second aerial vehicle 200 may be completely received within the fuselage 120.
[0138] The primary7aerial vehicle 102 may7be configured for flights over a distance, such as from a pickup location to the delivery7location. The secondary7vehicle 200 may be configured to carry or retain a payload. At a desired location, such as at a delivery7location, the secondary aerial vehicle 200 can be deployed from the primary aerial vehicle 102 to deliver a payload, as may be depicted in FIG. 2.
[0139] The primary7aerial vehicle 102 or the secondary aerial vehicle 200 may be autonomous, partially autonomous, or navigated at least in part by a user from a controlling location. The aerial vehicle system 100 may operate in rural or urban locations. Accordingly, the primary aerial vehicle 102 may deploy the secondary7aerial vehicle 200 in a variety of locations and environmental conditions.
[0140] Aerial Vehicle System Components
[0141] With reference to FIG. 3, an example schematic diagram of an example aerial vehicle system 100 is depicted. The aerial vehicle system 100 may include the primary aerial vehicle 102 and the secondary aerial vehicle 200. The aerial vehicle system 100 may include a fleet management system 114 in operative communication with the aerial vehicles 102, 200. The aerial vehicle system 100 may include a flight controller 116. The-20- 4910-9932-4237\3Docket Number P312830.W0.01 aerial vehicle system 100 may include a server 118 including or in operative communication with one or more databases or for executing one or more computing operations. The aerial vehicle system 100 may include or be in communication with a remote device 112, such as a user device or operator device.
[0142] The devices or components of the aerial vehicle system 100 may be communicatively or operatively coupled with each other, or with other devices for autonomous operation or navigation of the aerial vehicles 102. 200. For example, the aerial vehicle system 100 may include or be in operative communication with a network 110 through which the one or more components of the aerial vehicle system 100 may communicate. The network 110 may be or include one or more of, but not limited to a cellular network, Wi-Fi. Ethernet. Bluetooth, radio frequencies, or the like.
[0143] While reference may be made to a single device herein, it is appreciated that multiple of the same device may be included, such as multiple aerial vehicles 102, 200, multiple servers 118, or multiple flight controllers 116, and the like. Similarly, the operations of the various devices, such as the flight controller 116, remote device 112. server 118, or fleet management system 114 may be performed by a single device or unit, or by multiple separate or different devices distributed at one or more physical or virtual locations.
[0144] With reference to the primary aerial vehicle 102, the primary aerial vehicle 102 may include propulsion and flight control systems 140. The propulsion and flight control systems 140 may include a forward 142 and hover 152 propulsion assemblies described herein, or other features related to the flight or physical control of the primary aerial vehicle 102. The primary aerial vehicle 102 may include a retraction assembly 160 for selectively deploying and retracting the secondary aerial vehicle 200. For example, the retraction assembly 160 may be operatively connected with the tether 168 and retract or dispense the tether 168 to deploy or retract the secondary vehicle 200. The retraction assembly 160 may include a selective locking mechanism 162, or other features related to coupling or controlling the secondary' aerial vehicle 200 by the primary aerial vehicle 102.
[0145] With reference to the secondary aerial vehicle 200, the secondary aerial vehicle 200 may include propulsion systems 220 for independent movement of the secondary aerial vehicle 200 relative to the primary' aerial vehicle 102. For example, such-21- 4910-9932-4237\3Docket Number P312830.W0.01 propulsion systems 220 may be utilized for navigating to a payload location. The secondary aerial vehicle 200 may include payload release systems 230 providing the selective storage and release of pay loads for deliveries.
[0146] Either or both of the primary 102 and secondary aerial vehicle 200 may include communication systems 180, 250 for transferring information between the aerial vehicles 102, 200 or to one or more of the devices of the system 100. The communication systems 180, 250 may be wired and / or wireless systems. For example, the aerial vehicles 102, 200 may communicate over short or long ranges. In some examples, the aerial vehicles 102, 200 may be in communication with each other. In some examples, both or either vehicle 102, 200 may be in communication with a remotely located device, network 110, or other components of the system 100. In some examples, the aerial vehicles 102, 200 may communicate by radio frequencies, Bluetooth, Wi-Fi, cellular communication, satellite, or the like.
[0147] Either or both of the primary 102 and secondary' 200 aerial vehicles may include environmental sensors or sensor assemblies 170, 240, as described herein. The environmental sensors 170, 240 may capture, detect, or determine information about a surrounding environment or characteristics of the aerial vehicles 102, 200. The environmental sensors 170, 240 may include audio sensors, visual sensors, depth or distance sensors, or the like. The environmental sensors 170, 240 may include the various sensors that detect or determine force, acceleration, location, elevation, temperature, presence of objects, and / or similar characteristics of the aerial vehicles 102, 200 or the delivery environment.
[0148] The aerial vehicles 102, 200 may include navigation systems 190, 270. The navigations systems 190, 270 may determine locations, orientations, or movements of the aerial vehicles 102, 200. The navigation systems 190. 270 may receive information from the sensors 170, 240 or computing elements 185, 260 onboard or in communication with the aerial vehicles 102, 200. For example, the navigation systems 190, 270 may be in communication with the sensors 170, 240 to assist the aerial vehicles 102. 200 in determining the location or heading of the aerial vehicles 102, 200. The navigations systems 190, 270 may be in communication with global satellite systems (e g., GPS or GNSS) to determine delivery locations, conformity with flight paths, or the like. In some examples, information generated or received by navigation systems 190, 270 of one of-22- 4910-9932-4237\3Docket Number P312830.W0.01 the aerial vehicles 102, 200 may be communicated to the navigation systems 190, 270 of the other of the aerial vehicles 102, 200.
[0149] The aerial vehicles 102. 104 may include computing elements or resources 185.260. The computing elements 185, 260 may include one or more processors configured to receive information (such as from the on-board and off-board sensors) and utilize the information to make decisions, such as directing the aerial vehicles 102, 200 to fly to certain locations or the like. The computing elements 185, 260 may be operatively or communicatively coupled with the communication systems 180, 250. The computing elements 185, 260 can be located in whole or in part on the aerial vehicles, or at a secondary device or location. In some examples, the computing elements 185, 260 receive information from off-board sources, such as the flight controller 116, fleet management system 114, or other devices. In some examples, the computing elements 185, 260 may be operatively or communicatively coupled with the sensors 170, 240. The computing elements 185, 260 may interpret information gathered by the sensors 170, 240 to identify objects, features, or other characteristics of the surroundings. In some examples, the compute elements 185. 260 may interpret the information gathered by the sensors 170, 240 to determine a flight path or outputs by one or more components of the aerial vehicles 102, 200.
[0150] The various components of the aerial vehicles 102, 200, such as the computing elements 185, 260 may be varied based on the features type of aerial vehicle 102, 200, desired missions, and the like. For example, the functions of two or more of the components may be accomplished by a single device, or the functions of one or more of the components may be performed by two or more separate devices. As such, the discussion of any particular configuration for the aerial vehicles 102. 200 is meant as illustrative.
[0151] The fleet management system 114 may organize the distribution or operations of the aerial vehicles 102, 200. For example, the fleet management system 114 may track the assets or components of the aerial vehicle system 100 and execute logic to order operation of the system 100. The fleet management system 114 may determine or assign the aerial vehicles 102, 200 for various flight paths, delivery requests, service regions, or the like. During operation, the fleet management system 114 may receive delivery7requests and identify aerial vehicles 102, 200 to fulfill the delivery requests. The fleet-23- 4910-9932-4237\3Docket Number P312830.W0.01 management system 114 may determine landing, storage, or payload locations for the aerial vehicles 102, 200 to navigate between during a mission. In some examples, the fleet management system 114 may select an aerial vehicle 102. 200 from a plurality of aerial vehicles 102, 200 based on various payload capacities, performance characteristics, battery' status, or the like. Accordingly, the fleet management system 114 may issue commands to the aerial vehicles 102, 200 or other devices to perform various functions.
[0152] The flight controller 116 may generate or update flight paths for the aerial vehicles 102, 200. The flight controller 116 may generate flight paths in response to a delivery request from a remote device (e.g., remote device 112) or in response to a command from the fleet management system 114. Flight paths may define areas of space that are open or planned for navigation of an aerial vehicle 102, 200. In some examples, the flight paths include a time range or limit for traversing at least a portion of the flight path. Similarly, the flight controller 116 may define or identify regions or times where aerial navigation may be limited or prohibited. The flight controller 116 may determine flight paths between two or more locations, such as between a shipper or retail location and a delivery location. In some examples, the flight controller 116 may determine all aspects of the flight path. For example, the flight controller 116 may define a list of coordinates to navigate along or between. In other examples, the flight controller 116 may determine portions of the flight path, such as the start and end points or corridors open for travel by aerial vehicles 102, 200. In such an example, the aerial vehicles 102, 200 may determine or identify' the remaining portions of the flight path, such as autonomously. In some examples, the flight controller 116 may determine one or more maneuvers for an aerial vehicle 102, 200 to navigate a flight path, ft should be noted that although the flight controller 116 may generate general flight paths, the aerial vehicles 102, 200 may be configured to autonomously navigate along the path and avoid obstacles that may present themselves along the path, even if that means navigating out of the originally defined path. For example, the aerial vehicles 102, 200 may deviate from and return to the originally defined flight path, generate alternative flight paths, or portions of flights paths. During navigation of the flight paths the aerial vehicles 102, 200 may generate or determine autonomously at least some of the maneuvers to navigate the flight paths.-24- 4910-9932-4237\3Docket Number P312830.W0.01
[0153] The server 118 may be a computing system including one or more processing elements. The server 118 may include components for storing information or executing one or more operations for the system 100. For example, the server 118 may include or be in communication with a database of relevant information for aerial vehicle flights such as weather, map data, or various other ty pes of information on a delivery environment or delivery location, surrounding region, or the like. In some examples, as described herein, machine learned models may be used by the system 100 for autonomous or partially autonomous navigation of the aerial vehicles 102, 200. The machine learned models discussed herein or the training data for the models may be in communication with, located or stored at, or executed by the sen' er 118, or in communication with the various components by or through the server 118 or network 110.
[0154] The remote device 112 may be a user device or an operator device. A user device 112 may be operated by or receive inputs from a customer or end user of the system, such as a purchaser of a payload to be shipped by the system 100. An operator device may be operated by a developer or operator generating commands, or updates, to the functionality of the system 100. As an operator device, the remote device 112 may receive information from the system 100 or generate or communicate inputs or commands for communication to the system 100. For example, an operator or developer may use the remote device 112 to resolve conflicts in the system 100, define constraints or bounds of the system 100, correct or update flight paths, or various other tasks. As a user device, the remote device 112 may generate delivery' requests to the system 100 or provide other inputs to the system 100, such as by purchasing a product and designating a delivery method. The delivery requests may identify payloads 101 for delivery, or locations to receive and / or deliver payloads 101. The delivery requests may include information relating to the delivery such as a time to deliver or identify payload locations, e.g. areas preferable for payloads to be placed at, at the delivery' location. In some examples, a user device 112 may receive updates about the status or progress of a delivery request, such as an anticipated delivery time or date. A user device 112 may further receive and / or display for an end user other information about the system 100, such as real-time tracking information, status of systems or components of the primary aerial vehicle 102 and / or the secondary aerial vehicle 200, and the like.-25- 4910-9932-4237\3Docket Number P312830.W0.01
[0155] Primary and Secondary Aerial Vehicle Components
[0156] With continued reference to Figs. 1 and 2, the primary aerial vehicle 102 may include an aerodynamic fuselage or body 120 to assist in generating lift. The various components of the primary aerial vehicle 102 may be coupled to, extend from, or stored within the body 120. The primary' aerial vehicle 102 may include a vehicle compartment or bay 128 to store the secondary aerial vehicle 200 in the retracted or stowed position. The bay or vehicle compartment 128 may be defined by a recess or opening in a bottom 126 of the primary aerial vehicle 102.
[0157] The propulsion and flight control systems 140 of the primary' aerial vehicle 102 may include forward propulsion systems 142 for navigation over a distance. The primary' aerial vehicle 102 may include stationary or hover propulsion systems 152 for navigation over shorter distances, maintaining an aerial position, or for precise navigation.
[0158] The forward propulsion system 142 may include wings 144 and one or more forward rotor assemblies 148 to generate a forw ard thrust or lift for the primary' aerial vehicle 102. In some examples, the forward propulsion system includes a rear or tail propeller assembly 148 that may be coupled to the fuselage 120, such as to the tail 124. The forward rotor assemblies 148 may generate thrust sufficient to create lift by the wings 144. The forward propulsion system 142 may provide energy efficient flight over a distance.
[0159] In some configurations, the forward rotor assembly 148 can propel the aerial vehicle 102 in forward flight and / or hover or multi-dimensional flight. For example, the propeller assembly 148 may include two or more degrees of freedom that allow the propeller 150 to both transition betw een orientations for different modes of flight (e.g., forward flight and hover) and / or for the propeller 150 to be moved to compensate for a failure of another propeller (e.g., one of propeller assemblies 154). and / or adjust or account for environmental forces, such as wind gusts or the like.
[0160] The hover propulsion systems 152 may include one or more rotor assemblies 154 oriented to primarily provide a vertical lift for the primary aerial vehicle 102. The rotor assemblies 154 may include rotatable propellers to generate vertical lift. The hovering propulsion system 152 includes one or more booms 156 extending from the fuselage 120 and / or w ings 144 of the aerial vehicle 102 to support one or more propeller assemblies 154. In one embodiment, the one or more booms 156 are configured to-26- 4910-9932-4237\3Docket Number P312830.W0.01 position different propellers 154 assemblies at different locations relative to the fuselage 120 to help stabilize the vehicle 102. In some examples, such as where the forward rotor assembly 148 is articulable from a forward flight position to a second hover flight position, the forward rotor assembly 148 may be a part or portion of the hover propulsion systems 152.
[0161] The primary aerial vehicle 102 may include a docking assembly or fin 130. The fin 130 may be coupled to or extend from a top surface 122 of the fuselage 120. The fin 130 may be configured to be received or partially inserted within a dock for the aerial vehicle 102. The fin 130 may house or receive one or more components of the retraction assembly 160 for retracting or deploying the second aerial vehicle 104, as discussed in more detail below. The fin 130 may be located above the bay 128. In some examples, the fin 130 may be accessible from the bay 128.
[0162] As discussed herein, the primary aerial vehicle 102 and the secondary aerial vehicle 200 may be coupled by a tether or cable 168. A retraction assembly 160 may control the tether 168 to deploy, retract, or maneuver the secondary’ aerial vehicle 200 relative to the primary aerial vehicle 102. as shown, for example, in FIG. 5. The retraction assembly 160 may be contained within a fin 130 of the primary aerial vehicle 102. The retraction assembly 160 may include a drive mechanism 164 to selectively deploy, e.g., raise or lower, the secondary aerial vehicle 200 by the tether 168. For example, the tether 168 may extend into or out of the bay 128. The tether 168 and retraction assembly 160 may align with or be adjacent to the center of gravity of the primary aerial vehicle 102. By positioning the tether 168 at or near the center of gravity, a moment on the primary’ aerial vehicle 102 generated by the deployment or retraction of the secondary aerial vehicle 102 may be reduced. For example, the secondary aerial vehicle 200 and the primary aerial vehicle 102 may move relative to another without changing a balance of lift and gravitational forces.
[0163] With reference to Figs. 4 and 5, the tether 168 may be coupled to the secondary’ aerial vehicle 200 by a tether assembly 280. For example, the tether assembly 280 may be defined by or attached to the secondary aerial vehicle 200. The tether assembly 280 may correspond to a lock mechanism 162 to selectively couple or release the secondary vehicle 200 from the primary vehicle 102. The tether assembly 168 may include one or more electrical connections to provide a communication interface or electrical-27- 4910-9932-4237\3Docket Number P312830.W0.01 connection between the primary 102 and secondary 200 aerial vehicles. The electrical connection may be used to share power between the aerial vehicles 102, 200, such as from batteries 132, or for communication between the primary 102 and secondary aerial vehicle 200.
[0164] With reference to FIGS. 2 and4 the secondary aerial vehicle 200 may include a body 206 defining an interior pay load bay 210 to cany or receive payloads 101. The payloads 101 may be food, packages, or other items purchased or requested by a delivery recipient. The payload bay 210 may be selectively accessible or covered by a payload release assembly 230 including one or more movable doors 232. The doors 232 may be selectively closed or opened by an active feature such as a motor. The doors 232 may have a convex exterior or concave interior to move around the body 206.
[0165] The body 206 of secondary aerial vehicle 200 may have a corresponding shape to nest in the primary vehicle compartment 128. For example, the secondary aerial vehicle 200 may have a tapering shape, such as in the lengthwise or widthwise directions. When the secondary aerial vehicle 200 is stowed in the primary aerial vehicle 102. the bottom surface 208 of the secondary aerial vehicle 200 may align with the bottom surface 126 of the primary aerial vehicle 102 to define a single bottom surface. The tapering may assist in passive alignment of the secondary- aerial vehicle 200 with the bay 128 of the primary' aerial vehicle 102, such as by providing a greater tolerance when the secondary aerial vehicle 200 enters the bay 128. The body 206 may be arranged so that the center of gravity' of the secondary aerial vehicle 200 corresponds with an attachment location betyveen the secondary vehicle 200 and the primary vehicle 102, such as the tether 168. Aligning the center of gravities of the primary aerial vehicle and the secondary aerial vehicle 200 may provide increased maneuverability to the aerial vehicles 102, 200. or dunng deployment of the secondary aerial vehicle 200.
[0166] The secondary aerial vehicle 200 may include one or more propulsion assemblies 220 to maneuver relative to the primary aerial vehicle 102. The assemblies 220 generate thrust to move the secondary' aerial vehicle 200 in two or more directions. A rear propulsion assembly 226 may provide forward or rearward thrust relative to the secondary aerial vehicle 200. A fore thruster assembly 222 or an aft thruster assembly 224 may be located to the fore or aft, respectfully, of the payload bay 210. The fore 222 or aft 224 thruster assemblies may be oriented to generate side thrust, e.g., rightward or-28- 4910-9932-4237\3Docket Number P312830.W0.01 leftward thrust, relative to the secondary' aerial vehicle 200. In operation, the propulsion assembly 220 may be used by the secondary aerial vehicle 200 to generate holonomic thrust. For example, the secondary aerial vehicle 200 may generate thrust to move forward, rearward, left, right, or rotate relative to the primary aerial vehicle 102. By arranging the propulsion assembly 220 to provide thrust in more than one direction simultaneously, the secondary vehicle 200 may move in multiple directions irrespective of orientation, enabling changes in velocity or direction without first reorienting the vehicle 200.
[0167] In operation, the secondary aerial vehicle 200 may be deployed to a position at or adjacent a delivery surface. For example, the secondary aerial vehicle 200 may land on the delivery’ surface. Extensions from the bottom, or feet 212, may provide support or provide a spacing betyveen the delivery surface and the doors 232 to release a payload 101. In other examples, the secondary aerial vehicle 200 may be suspended by the tether 168 above the delivery surface during release of the pay load 101.
[0168] With reference to FIG. 4, either or both the primary 102 and secondary aerial vehicle 200 may include a power source 132 to provide power to the active components of the aerial vehicles. The poyver source 132 may be a battery, fuel, or the like. The poyver sources 132 of either or both the primary aerial vehicle 102 or the secondary- aerial vehicle 200 may provide power to the other or both aerial vehicles 102, 200.
[0169] The aerial vehicles 102. 200 may include one or more safety features. For example, either or both aerial vehicle 102, 200 may include a parachute or similar recovery' mechanism 136. Similarly, a payload may be attached to a parachute or similar feature to assist in a controlled descent. The secondary aerial vehicle 200 and, or primary’ aerial vehicle 102 may further include a line cutter or another device to selectively disconnect the secondary aerial vehicle 200 from the primary aerial vehicle 102.
[0170] Sensors and Navigation Components
[0171] With continued reference to FIGS. 1-5 and additional reference to FIGS. 6-7, the aerial vehicles 102, 200 may include one or more sensors or sensor assemblies 170, 240 to collect data or assist in the operation or navigation of the aerial vehicles 102, 200. For example, the aerial vehicles 102, 200 may be operated remotely or autonomously by detecting information regarding a delivery environment, such as obstacles, yveather information, and the like, and / or information regarding the aerial vehicles 102, 200 such-29- 4910-9932-4237\3Docket Number P312830.W0.01 as a location or position, orientation, velocity, or the like. The gathered information may be analyzed to determine or update movements of the aerial vehicles 102, 200. To analyze or process information gathered by the sensors 170, 240 the aerial vehicles 102.200 may include or be in communication with one or more computing element 185, 260 including a processing element. In some examples, the computing element 185, 200 may be or include multiple processing elements and / or multiple controllers. The computing element 185, 260 may be used by or operatively associated with the aerial vehicles 102.200 to determine or execute operations regarding flight, docking, and / or landing, e.g., execute of software or other instructions that assists the vehicles 102, 200 in executing on decisions and flight operations.
[0172] While specific examples and combinations of sensors are described herein, it is appreciated that a variety of t pes of sensors and positions of sensors may be used. The sensors may be positioned to capture information about a navigation or delivery environment in one or more directions relative to the aerial vehicles 102, 200. Further, some or all of the sensors may be located at each of the aerial vehicles 102, 200 or distributed between the aerial vehicles 102. 200.
[0173] Primary Vehicle Sensors
[0174] The primary' aerial vehicle 102 includes one or more sensors or sensor assemblies 170. In some examples, the primary aerial vehicle 102 may include a plurality of sensor assembles 170. The sensor assemblies 170 may be coupled to or housed within the fuselage 120 and / or wing assembly 146. The sensor assemblies 170 may detect or include components to detect various information about the delivery environment or the aerial vehicles 102, 200. In one embodiment, the sensor assemblies 170 may include a sensor and a probe or support structure that attaches the sensor to the aerial vehicle 102 or spaces the sensor assemblies 170 from the aerial vehicle 1 2.
[0175] The sensors 170 may include acoustic or audio sensors 172. The audio sensors 172 may be configured to detect noise, sound waves, or vibrations of an environment or the aerial vehicles 102, 200. For example, the acoustic sensor 172 may include one or more microphones, piezoelectric sensors, or ultrasonic sensors. The audio sensors 172 may be spaced over the primary aerial vehicle 102 to assist in triangulating or determining the source or direction of an audio emission.-30- 4910-9932-4237\3Docket Number P312830.W0.01
[0176] The sensors 170 may include visual sensors or light sensors 174. The visual sensors 174 may be one or more cameras or light detecting elements. The camera 174 may capture images continuously (e.g.. a video or rolling shutter), over predetermined intervals, or at discrete times, such as in response to a detected condition. In one example, the camera 174 of the primary aerial vehicle 102 may be oriented to view forward, or generally in the direction of travel. In some examples, two or more cameras 174 may be used. For example, a first camera 174 may look forward or towards a direction of travel, while a second camera 174 is positioned to capture visual data below or above the primary aerial vehicle 102, such as towards the ground or below horizon regions of the delivery environment. In some examples, two or more cameras 174 may be used together to determine depth or distances of images, e.g., act as a stereo camera system. For example, two spaced cameras 174 may capture different perspectives of the same objects as a result of the physical spacing which can be used to determine a distance of captured environmental features. In some examples, the primary aerial vehicle 102 may include one or more light sources to assist in illuminating the delivery environment or assist the visual sensors 174 in capturing images or data.
[0177] The sensors 170 may include depth or distance estimating sensors. The depth or distance estimating sensors may capture information regarding or determine distances between the aerial vehicles 102, 200 such as objects, structures, other aerial objects, the ground, or the like. In some examples, the visual sensors 174 or audio sensors 172 may be used to determine distances. In some examples, various sensors for depth collection can be used, e.g., LiDAR, direct depth cameras, texture projectors, radar, sonar, a monocular camera depth estimator, or the like. As such, the discussion of any particular sensor is meant as illustrative only. The machine learned models, as discussed herein, may be used to determine, verily, or assist the sensors 170 in determining depths or distances relative to the aerial vehicles 102, 200.
[0178] In some examples, the primary aerial vehicle 102 includes sensors 170 that may detect various factors, conditions, or statuses of an environment or the aerial vehicles 102, 200. For example, the sensors 170 may detect a force, acceleration, location, elevation, temperature, and / or similar characteristics of the aerial vehicles 102, 200 or of the delivery environment. For example, the sensors 170 may determine a wind velocity relative to the aerial vehicles 102, 200 (e.g., an anemometer). The sensors 170-31- 4910-9932-4237\3Docket Number P312830.W0.01 may communicate with or receive satellite or cellular information to determine a geographic position or location of the aerial vehicles 102, 200 (e.g., include a global positioning sensor (GPS) sensor). The primary aerial vehicle 102 may include an altimeter or another device to determine an elevation or a relative distance between the aerial vehicle 102 and the ground or objects. In some examples, the sensors 170 may detect a change in an electrical current resistance or heat exchange rate corresponding to an external temperature, precipitation, or ice on the aerial vehicles 102, 200. e.g., include one or more thermal sensors, humidity sensors, or the like. It should be noted that the sensors 170 may be configured to directly detect certain characteristics (e.g., heat) and / or the characteristics can be determined based on changes to a detected characteristic.
[0179] In some examples, a sensor 170 may be coupled to or associated with the tether 168 or retraction assembly 160. In one example, a sensor 170 may determine a tension of the tether 168 or a torque of, or forces on, the retraction assembly 160 corresponding to holding, releasing, or retracting the tether 168. The tension on the tether 168 or forces on the retraction assembly 160 may indicate a weight of the secondary aerial vehicle 200 or pay load 101, a velocity of ascent or descent of the secondary aerial vehicle 200, or drag forces (e.g. due to wind) on the tether 168 or secondary aerial vehicle 200 during deployment.
[0180] Secondary Aerial Vehicle Sensors
[0181] With reference to FIG. 7, the secondary aerial vehicle 200 may include one or more sensors or sensor assemblies 240. The sensors 240 may be coupled to an exterior of the secondary aerial vehicle 200 or housed within a portion of the body 206 of the secondary aerial vehicle 200. For example, the sensor assembly 240 may be located on the bottom 223 of the secondary aerial vehicle 200. The sensors 240 of the secondary aerial vehicle 200 may capture or determine various information about the delivery environment and / or the aerial vehicles 102, 200. In some examples, the sensor assembly 240 may detect or interpret information while stowed in the primary aerial vehicle 102 or during deployment from the primary' aerial vehicle 102. For example, by locating at least some of the sensor assemblies 240 on the bottom of the secondary aerial vehicle 200, the sensors 240 may be selectively operated by or accessed when the secondary aerial vehicle 200 is stowed in the primary aerial vehicle 102. For example, the primary aerial-32- 4910-9932-4237\3Docket Number P312830.W0.01 vehicle 102, or aerial vehicle system 100 generally, may operate the sensors 240 while the secondary aerial vehicle 200 is carried or stowed by the primary aerial vehicle 102.
[0182] The sensors 240 may include similar functionality and may detect similar characteristics as sensors 170 and / or may detect different characteristics. In some embodiments, sensors 170 of the first aerial vehicle 102 and sensors 240 of the second aerial vehicle 200 may include data that is used together to make determinations of various characteristics. For example, the data or captured information of one aerial vehicle 102, 200 may be combined, compared, or referenced together with the data or captured information of the other aerial vehicle 102, 200.
[0183] The sensor assembly 240 of the secondary aerial vehicle 200 may include one or more depth sensors or devices 245 to determine distances to objects relative to the aerial vehicle 200. In some examples, the depth sensor 245 may be a three-dimensional camera (e.g., camera or assembly capturing two or more images to determine depth) or other depth or distance detection component. For example, the depth sensor 245 may be one or more of cameras. LiDAR, radar, telemeter, sonar, direct depth cameras, texture projectors, a monocular camera depth estimator, or various other devices for determining or capturing information indicative of a distance between two points. The three-dimensional information may be used to navigate to the deli very location or a tether 168 dispensing rate or speed to deploy secondary aerial vehicle 200 from the primary aerial vehicle 102.
[0184] The depth sensors 245 may be a pair of two-dimensional stereo cameras 244 used in combination to determine depth. The stereo cameras 244 may be arranged facing downw ards relative to the secondary aerial vehicle 200. The stereo cameras 244 may be spaced by a known distance and orientation relative to the secondary aerial vehicle 200. The stereo cameras 244 may each capture two-dimensional images of the surrounding environment. The data of each image may be compared or combined, based on the known spacing between and the orientation of the stereo cameras 244, to determine three-dimensional characteristics of the surrounding environment. For example, the stereo cameras 244 may be used to determine a distance to an object or a distance to a delivery location.
[0185] The sensor assembly 240 may include a monocular camera 242. The monocular camera 242 may be positioned to face downwards relative to the secondary-33-4910-9932-4237\3Docket Number P312830.W0.01 aerial vehicle 200. The monocular camera 242 may also or similarly be used to capture images and determine characteristics of the surrounding environment. The monocular camera 242 may, in some examples, include a wide-angle lens to increase a field of view of the sensor assembly 240. For example, the lens 242 may provide a view of the delivery location or area surrounding the feet 212 of the secondary' aerial vehicle 200 when on the landing surface to determine a presence of any objects. In some examples, a machine learned model or additional sensors 240, such as LiDAR, may be used in combination with the monocular camera 242 to determine or estimate distances to features captured in images of the monocular camera 242. Accordingly, the monocular camera 242 may act as a depth sensor 245.
[0186] In operation, the cameras 242, 244 may capture images or other visual data. The camera or cameras 242, 244 may capture images continuously, at regular intervals, or discrete images. The images may be captured automatically, in response to a stimulus or condition, or at predetermined intervals of time or distance.
[0187] In some examples, the cameras 242, 244 may capture a variety of spectrum of lights. In some examples, the sensors 240 capturing light information may include a filter to enhance or limit light captured by the sensors 240. The filters may be physical or digital. For example, the physical filters may be lens or coverings on the cameras 242 or 244, or the filtering may be performed by one or more operations executed by processing elements to identify various characteristics or wavelengths of light captured by the sensors 240. For example, the cameras 240 may be provided with a digital or physical filter (e.g., a dual band pass filter placed over a lens of the camera) for image capture when illumination is provided in a variety of spectrums. In some examples, the cameras 240 may collect images in both the visible light spectrum (e.g.. during the day) and the 1R spectrum (e.g.. when active illumination is used at night).
[0188] The sensor assembly 240 may include a light generating element or light source 246. The light source 246 may be located adjacent the one or more cameras or sensors. The light source 246 may provide illumination to improve the capturing of images by the cameras. The light source 246 may include one or more LEDs to generate electromagnetic radiation. The light source 246 may generate a variety of wavelengths of electromagnetic radiation. In one example, the light source 246 may generate light in the infrared (IR) spectrum (e.g., around 815nm wavelength). IR light is not visible to-34- 4910-9932-4237\3Docket Number P312830.W0.01 humans and may provide a less distracting or intrusive source of illumination for the sensor assembly 240. IR illumination may be outside of the visible spectrum, such that the aerial vehicle 102. 200 may deliver payload 101 at night without visible lighting that may disturb humans close to the delivery area. The illumination may assist the sensors 170, 240 in capturing visual data or images of the environment within a certain distance from the aerial vehicles 102, 200 (e.g., a radius of 10m). In some examples, the light source 246 may generate structured light and be situated a known distance from the one or more cameras 240. The structured light and known distance from the one or more cameras 240 may assist the secondary' aerial vehicle 200 in determining the distance or presence of objects.
[0189] The secondary aerial vehicle 200 sensors 240 may include audio sensors. For example, the audio sensors may capture sound or vibrations from a delivery environment or from the aerial vehicles. The audio sensors may be positioned within or coupled to the secondary vehicle 200.
[0190] The information captured by the sensor assembly 240 may be used to detect or determine the presence or location of objects, such as along a flight path or at a delivery location. For example, the visual information captured by the cameras 242, 244 may be used to identify objects or features of an environment. In some examples, differences in determined depths or distances to features may be used. Capturing infrared light may assist in identifying thermal characteristics of objects, such as differences in temperatures between objects within the delivery environment. The thermal characteristics may be indicative of some objects, such as by the type of object, or for distinguishing between or boundaries of objects. For example, a cable (e.g., telephone, power line, lights) or a structure (e.g.. garage or house) may have an increased temperature compared to a surrounding environment. Similarly, objects (e.g., furniture, vehicles, pavement) in an outdoor area (e g., field, park, yard) may have an increased or decreased temperature compared to natural elements such as trees, grass, or dirt.
[0191] The secondary aerial vehicle 200 may include additional sensors 240 that may detect a variety of additional characteristics of the delivery environment or the aerial vehicles 102, 200. For example, the sensor assembly 240 may detect characteristics such as objects, altitude, temperature, wind speed or direction, location or positions of the aerial vehicles, or movements or forces experienced by aerial vehicles. In one example,-35- 4910-9932-4237\3Docket Number P312830.W0.01 the secondary aerial vehicle 200 may include an altimeter or another device to determine an elevation of the secondary vehicle 200. In some examples, the sensors 240 may communicate with, e.g.. send or receive, satellite or cellular information to determine a geographic position or location of the aerial vehicles 102, 200.
[0192] In some examples, the information gathered by the sensors 240 of the secondary aerial vehicle 200 may be received by or communicated to the primary aerial vehicle 102. In some examples, the information gathered by the sensors 170 of the primary aerial vehicle 102 may be received by or communicated to the secondary aerial vehicle 200.
[0193] Overview of Operation of the Aerial Vehicle System
[0194] During operation of the aerial vehicle system 100, the aerial vehicle system 100 may transport a payload 101 from or traverse between a first location to a second location. The aerial vehicle 102 may transport the secondary aerial vehicle 200 carrying the payload 101 between the locations. Either or both locations may be delivery7or pickup locations for the payload 101. For example, the first or second locations may be warehouses, restaurants, retail stores, service centers, residential locations, businesses, or similar locations where delivery7services may be utilized. In some examples, the locations may include landing or takeoff structures designed to support or connect with the aerial vehicles 102, 200. For example, the structures may be landing pads, suspended features, or docking structures. In some examples, a single location may include multiple landing structures. For example, a shipper may include a plurality of structures to receive or support aerial vehicles 102, 200.
[0195] The operation of the system 100 may begin responsive to a delivery7request. For example, a delivery request may be received from a remote device 112 or other device in communication with the aerial vehicle system 100. The delivery request may identify a payload to be delivered by the aerial vehicle system 100. The delivery' request may specify a ty pe of a payload and a delivery location. The delivery' request may specify where the payload is currently located or where the payload is to be delivered from, such as a retailer or business. The delivery request may identify a region or payload location at the delivery location to deliver the payload or a landing area for the aerial vehicles 102, 200. The payload location may be a specific portion of a building or area, such as a door, window, deck or patio, roof, parking area, yard, or other locations-36- 4910-9932-4237\3Docket Number P312830.W0.01 accessible by a delivery recipient or the aerial vehicles 102, 200. The delivery request may be received from a remote device 112, such as by the network 110.
[0196] After receiving the delivery request, the fleet management system 114 may identify one or more aerial vehicles 102, 200 to complete the delivery request or mission. The aerial vehicles 102, 200 may be selected based on a proximity to either the first or second locations, or performance characteristics of the aerial vehicles 102, 200 such as a payload capacity or battery status. In some examples, the fleet management system 114 may identify a shipping location or payload to be shipped, such as when the good is available at multiple separate retailers or locations of a retailer.
[0197] The flight controller 116 may generate one or more flight paths, or portions of flight paths, for the aerial vehicles 102, 200 responsive to the delivery request. The flight path may be to the shipping location to receive the payload or from the shipping location to the delivery location. In some examples, the flight controller 116 may update the flight paths during navigation of an existing flight path.
[0198] At the first location, the secondary’ aerial vehicle 200 may be presented to receive a payload 101 or release the payload 101. The payload 101 may be securely stored in or coupled to the secondary vehicle 200. For example, the payload 101 may be positioned at least partially in the payload bay 210. During loading, the secondary’ vehicle 200 may be deployed from or coupled to the primary vehicle 102. After loading the secondary aerial vehicle 200, the secondary aerial vehicle 200 may be retracted to the primary aerial vehicle 102. In some examples, the aerial vehicle 102 may receive the payload 101 at the second location or exchange payloads 101 at the second location.
[0199] The primary' aerial vehicle 102 may carry' the secondary aerial vehicle 200 and the payload 101 to the second location or delivery’ location. The primary aerial vehicle 102 may navigate along the flight path at least partially autonomously. For example, the aerial vehicles 102, 200 may follow a defined route or determine a route between the locations. The sensors 170, 240 of the aerial vehicles 102, 200 may gather information about the environment of the first or second locations or along the flight path to assist in navigating the aerial vehicles 102, 200. As described herein with particular reference to Fig. 8, computing elements 185, 260 or machine learned models may be utilized to interpret information gathered by the sensors 170, 240 to determine compliance with the flight paths, such as the location of the vehicles 102, 200, or to determine various-37- 4910-9932-4237\3Docket Number P312830.W0.01 maneuvers. In some examples, the computing elements 185, 260 or machine learned models may autonomously generate updated or new flight paths based on information gathered by the sensors 170, 240.
[0200] At the second location the primary aerial vehicle 102 may transition from forward or traversing flight to hovering or stationary flight. In some examples, the primary aerial vehicle 102 may dock or otherwise connect to a structure at the second location. The second or delivery location may be identified by information captured by the sensors 170, 240. After or during identification of the delivery location, the payload location or alternative locations to deliver the payload 101 may be determined. For example, the sensors 170, 240 may identify objects or areas of an environment. As described herein, the sensors 170. 240 may identify regions where the primary 102 or secondary aerial vehicle 200 may navigate or where the secondary aerial vehicle 200 may land, such as a payload location.
[0201] After reaching or identifying the second location (e.g. a delivery location or payload location) the secondary aerial vehicle 200 may be deployed from the primary aerial vehicle 102. To deploy the secondary aerial vehicle 200. the tether 168 may be selectively released or extended from the retraction assembly 160. For example, the retraction assembly 160 may dispense tether 168 for the secondary' aerial vehicle 200 to descend. Accordingly, the retraction assembly 160 or the primary' aerial vehicle 102 may control vertical movement of the secondary vehicle 200. The propulsion assembly 220 may provide independent movement or control for the secondary aerial vehicle 200 during deployment. For example, the propulsion assembly 220 may counter disturbances, such as wind, by generating corresponding or opposite thrust. Additionally, or alternatively, the propulsion assembly 220 may adjust for positional errors or environmental factors, such as wind or objects, and navigate to the delivery location. In this manner, the first aerial vehicle 102 may control large movements of the second aerial vehicle 200, while the second aerial vehicle 200 may have an ability' to move relative to the first aerial vehicle 102. The movement of the secondary vehicle 200 may be ultimately restricted, such as by the tether or cable 168. In other embodiments, the second vehicle 200 may be able to move more freely relative to the first aerial vehicle 102 or may move less freely, e.g., not include separate propulsion.-38- 4910-9932-4237\3Docket Number P312830.W0.01
[0202] During operation or flight, the aerial vehicles 102, 200 may perform maneuvers or actions to assist in collecting information by the sensors 170, 240. The maneuvers may be predetermined, responsive to received information during navigation, or autonomously generated during flight. The maneuvers may be to detect or avoid an object, or to gather additional information about a surrounding environment or of the aerial vehicles 102, 200. For example, a maneuver may be initiated when information captured by the sensors 170, 240 is unclear or has a low confidence rating. To collect additional information, the aerial vehicles 102, 200 may maneuver to additional or various perspectives of a regions, such as a delivery location or along a flight path. In some examples, either or both of the aerial vehicles 102, 200 may move relative to the delivery location to capture a height or depth of an object. The maneuvers may include translational (e.g., side to side) movements relative to an area or flight path, or rotation of the aerial vehicles 102, 200 to increase a field of view or orientations of the sensors. The maneuvers may assist in determining three-dimensional perspectives. For example, in some perspectives, elongated objects (e.g., a banner or a pole) may be appear similar to a thinner or shorter object (e.g.. a line or a ball) prior to determining a depth or height of the object or feature. The different positions may be used to confirm or capture the depth to better characterize the objects.
[0203] The action or maneuvers may be used to surveil for, detect, or locate moving objects, whether ground based or airborne. For example, as may be shown in FIG. 2, the primary aerial vehicle 102 may conduct a “light housing’’ maneuver to rotate 108 about an axis 106 at the deliver}' location, or otherwise while hovering. Such a maneuver may help to identify features within the environment in a more holistic manner than may be detected with other techniques (e.g., helps to avoid blind spots with the sensors and calibrate the sensors to the lighting and other aspects of the environment). In some examples, the primary aerial vehicle 102 may rotate 108 before, during, or after deployment of the secondary aerial vehicle 200. By rotating, the primary aerial vehicle 102 may survey the area around the aerial vehicles 102, 200 using the visual sensors. The rotation may act as a periodic surveillance of an area over a period of a time to determine if objects have or are moving or changing a position. The lighthouse maneuver may also help to provide a full view' or substantially full view' of the environment without having to rotate or otherwise move the visual sensors. For example, by relying on the flying-39- 4910-9932-4237\3Docket Number P312830.W0.01 capabilities of the aircraft, any occlusions in the sensor viewing angles or the like can be eliminated or reduced. This helps to reduce moving parts (which can increase reliability) as well as reduce the number of sensors needed by the aircraft, while still providing sufficient visual coverage). In examples where the aerial vehicles 102, 200 include fixed cameras or other sensors, the fixed orientation and position of the sensors can assist in determining environmental information (e.g., position and orientation of objects relative to the aerial vehicle) because the sensors will have a known and consistent distance and orientation relative to other features of the aerial vehicles 102, 200. For example, a movable sensor can introduce additional variables, or processing demands, to determine the new orientation or offset of sensors relative to other features of the aerial vehicle. The lighthouse maneuver can be activated at various times during the mission, e.g., right after takeover, during flight, or before and after delivery.
[0204] During the rotation of the primary aerial vehicle 102, 200 the sensors 170, 240 may triangulate or determine the position of an object or emission of light or sound by¬ comparing intensities of light or sound at different orientations or time. The coupling of the secondary aerial vehicle 200 to the primary aerial vehicle 102 (e.g., the position of the tether 168) at or adjacent the center of gravities of the respective aerial vehicles 102, 200 may facilitate independent movement of the vehicles 102, 200 while the other performs a maneuver. For example, coupling the tether 168 at the center of gravities maylimit or reduce a moment about the tether 168 resulting from a maneuver of the aerial vehicles 102, 200, providing increased stability for the aerial vehicles 102, 200 during such maneuvers.
[0205] In some examples, the payload location or delivery location may be identified or validated after or during deployment of the secondary aerial vehicle 200. For example, during deployment the sensors 170. 240 may capture information, such as images, of the delivery environment. Various features may be identified within the deliveryenvironment, such as objects or obstacles, by the system 100. The identified features may satisfy conditions for a payload location, or correspond to a known or requested pay load location. In some examples, the sensors 240 of the secondary aerial vehicle 200 may capture additional information during descent to update, add to, or confirm the presence of the identified features.-40- 4910-9932-4237\3Docket Number P312830.W0.01
[0206] At the delivery location, the secondary7aerial vehicle 200 may land on or descend to a delivery surface at the payload location. Depending on the payload 101 or payload packaging, the payload 101 may be released onto or above the payload location, such as when the potential energy of the pay load 101 is sufficient to prevent damage. In some examples, the pay load 101 may include packaging, such as a box, padding, or a parachute. The pay load 101 packaging may be designed to slow a descent, disperse impact forces, or limit movement of the pay load 101 after release. To reach the open configuration or release the payload 101, one or more doors 232 of the pay load release assembly 230 may open to expose the payload bay 210. In the open configuration, the payload 101 may exit the payload bay 210. After release, the secondary7aerial vehicle 200 may be lifted off or away from the payload location or the delivery7surface.
[0207] After the delivery, the secondary aerial vehicle 200 may be retracted to the primary aerial vehicle 102. After the secondary' aerial vehicle 200 is coupled to the primary7aerial vehicle 102, the aerial vehicle system 100 may return to the first location, such as a shipping location or docking area. In some examples, the aerial vehicle system 100 may initiate a new mission or flight path upon completion of the initial mission.
[0208] The sensors 170, 240 may be used for delivery verification. For example, after a payload 101 is released from the secondary aerial vehicle 200, an image sensor (e.g., the stereo cameras 244 or the monocular lens 242) may capture an image or otherwise detect the presence of the payload 101 outside the secondary aerial vehicle 200 at the delivery location. The secondary aerial vehicle 200 may be raised above the delivery' location and an image captured to verily release or delivery7of a payload 101. In some examples, one or more sensors 170, 240 may detect or determine a current tension or change in tension on the tether 168. For example, a before release tension and an after release tension may be compared (e.g. a change in tension) to a known or estimated payload weight to determine whether the payload 101 has been released.
[0209] The various operations of the aerial vehicle system 100 may be accomplished autonomously or partially autonomously. For example, as described herein the aerial vehicle system 100 may include or be in communication with one or more machine learned models to perform the actions. Further, as described herein, the aerial vehicles 102, 200 may perform various maneuvers to assist in gathering information or performing the operations.-41- 4910-9932-4237\3Docket Number P312830.W0.01
[0210] Autonomous Modeling and Control
[0211] FIG. 8 is a schematic diagram of a process flow of information through and communication between various example compute modules or resources 1020 of the aerial vehicle system 100 for navigating through an environment. The compute modules or models 1020 are illustrative and the various functionality or features described herein may be distributed across additional resources, shared between the described resources, or be components of a larger combined resource.
[0212] The compute resources 1020 may provide functionality to navigate the aerial vehicles 102, 200 through various components for implementing perception scene understanding and planning within an environment to navigate the aerial vehicles 102, 200. The perception scene understanding and planning (e.g. autonomous navigation systems) may be executed by one or more compute resources 1020 for the aerial vehicles 102, 200 or system 100 to autonomously or independently identify objects or features of the environment or positions of the aerial vehicles 102, 200 within the environment. The perception scene understanding and planning may be used by the aerial vehicles 102, 200. or aerial vehicle system 100. to generate flight paths, navigate around obstacles, deliver to precise locations such as under overhangs and awnings, or determine a delivery location or payload locations to place a payload 101. The delivery’ location or delivery area may be a geographically defined region, such as an address, coordinates, or the like, where a payload 101 or the aerial vehicles 102, 200 are requested. A payload location may be a portion of the delivery area defined by environmental features or objects where a payload 101 may be delivered or an aerial vehicle may cease flight.
[0213] In some examples, the perception scene understanding and planning may provide the aerial vehicle 102, 200 with, or generate information based on captured information by one or more sensors 1040 representative of depth information for an environment. The perception scene understanding may augment, analyze, combine or fuse information captured or generated by sensors 1040 (such as GPS or positional coordinates, captured images, depth or distance information) to deliver a payload or to ascertain position of aerial vehicles 102, 200 within the environment. For example, the aerial vehicle system 100 including the compute resources 1020 may identify and determine dimensions of various features or objects such as electrical wires, fences, and the like.-42- 4910-9932-4237\3Docket Number P312830.W0.01
[0214] The compute resources 1020 may be located at or in communication with one or more of the various devices of the aerial vehicle system 100. For example, the compute resources 1020 may be distributed across or executed by one or more of the aerial vehicles 102, 200, the server 118, the flight controller 116, fleet management system 114, remote device 112, or the other devices. In some examples, the compute modules 1020 may be executed by or stored on some or all of the compute elements 185, 260 of the aerial vehicles 102. 200. The compute resources 1020 may include processing or memory resources. For example, memory resources may store instructions which, when executed by the processing elements implement the functionality of the modules as exemplified in FIG. 8.
[0215] The compute resources 1020 may include, execute, or communicate with various machine learning and / or artificial intelligence (ML or Al) models. It should be noted that the various ML or Al models of the compute resources 1020 may be trained on different or separate computing or processing elements and then deployed to the aerial vehicle system 100. For example, the various machine learned models may be stored or deployed on board the aerial vehicles 102, 200. In some examples, the machine learned models or other compute resources 1020 may be located in a cloud or distributed network, or at another networked processing element that are accessible by the aerial vehicles 102, 200.
[0216] While FIG. 8 may show an example order to the flow or generation of information, it is appreciated alternative arrangements may be utilized as understood by one of skill in the art. In some examples, the functions of one or more of the compute resources 1020 may be separated into two or more separate compute resources 1020, or additionally or alternatively, the functions of at least a portion of two or more of the compute resources 1020 may be combined or performed by a single or different compute resources 1020.
[0217] The compute resources 1020 may receive various information from sensors 1040 of one or both of the aerial vehicles 102, 200 to determine information about or actions responsive to features of an environment. The sensors 1040 may be similar to or include the sensors 170, 240 discussed herein, such as with reference to the primary aerial vehicle 102 or the secondary aerial vehicle 200. For example, the sensors 1040 may include positional sensors to capture or generate information related to a global-43- 4910-9932-4237\3Docket Number P312830.W0.01 navigation satellite system (GNSS) or position, location, altitude, or related types of information. The sensors 1040 may include or represent sensors capturing or detecting information related to velocity, acceleration, forces or other physical characteristics of the aerial vehicles 102, 200. The sensors 1040 may include depth or distance detecting sensors. The depth detecting sensors may be visual or electromagnetic sensors such as Lidar, monocular or wide-angle cameras, and stereo cameras, or the like. Some sensors 1040 may be various environmental sensors for detecting or determining information such as audio sensors, temperature, wind speeds, time of day, or other relevant factors to navigating or operating the aerial vehicles 102, 200. It is appreciated that the sensors 1040 may include additional or alternative sensing elements to receive or capture additional information as described herein.
[0218] The compute resources 1020 may generate or communicate outputs or information to one or more systems or components of the aerial vehicle 102, 200 for autonomous operation of the aerial vehicles 102, 200, such as guidance, navigation, or control (GNC) systems 1200. The GNC systems 1200 may include or represent the features of the aerial vehicles 102, 200 responsible for flight, maneuvers, or other physical operations. For example, the GNC systems 1200 may include or be the propulsions systems 140, 220, the retraction assembly 160, payload release assembly 230, or the like. GNC 1200 may use information from the compute resources 1020, such as planned trajectories or instructions for maneuvering the aerial vehicle 102, 200. to generate or execute commands to the components of the aerial vehicles 102, 200 to navigate through an environment.
[0219] Turning to the compute resources 1020. the compute resources 1020 interpret or identify information captured by sensors 1040 or information received from other compute resources 1020. For example, machine learned models may receive data and other inputs associated with the sensors 1040 to interpret or generate new or additional information for operations by the system 100 or another machine learned model or compute resource 1020. The various compute resources or ML models 1020 may interpret the images to produce image data or model the environment around the aerial vehicles 102, 200.
[0220] The compute resources 1020 may determine or identify characteristics of the environment, such as depth or distance information or identities or shapes of objects,-44- 4910-9932-4237\3Docket Number P312830.W0.01 within information or data captured by the sensors 1040. The compute resources 1020 can identify payload locations or landable areas, or routes through the environment based on the captured information. In some examples, a single model can identify routes or landable zones based on environmental characteristics from the captured data, or two or more models can separately identify environmental characteristics useful in modeling or understanding the environment.
[0221] In some examples, the compute resources 1020 include a landability model 1100 trained on data representative of depth or object classifications to recognize landable zones or to model the environments around an aerial vehicle for navigation. In one example, the landability model 1100 analyzes image data to three dimensionally model or identify regions within a three dimensional environment suitable for navigation or landing by an aerial vehicle.
[0222] As illustrated in Fig. 8, the landability model 1100 can be a single model, or the functions of the landability model 1100 can be distributed between two or more separate models. Accordingly, while description of various machine learned models are described herein, such as a depth estimation model 1160 to determine distances or a semantic understanding model 1180 to individually identify objects or object types, it is appreciated that the functionality of these models can be combined or integrated into a single model (e.g. the landability model 1100) and specific examples of separate models, or their functionality, are for illustrative purposes.
[0223] For example, while specific outputs are described with reference to individual models (e.g. the semantic understanding model 1180, depth estimation model 1160, the 3D world modeling model 1280, or the like) it is appreciated that such outputs are exemplary and in a single or integrated model such outputs can be optional. For example, with reference to the depth estimation model 1160. in some examples pixels of images can be associated with distances and provided as an output to a separate world model 1280 to model the environment. In other examples, a single landability model 1100 trained on determining distances and three dimensional modeling can model the environment of an aerial vehicle, such as to map or model the environment or to identify landable zones based on elevations of the environment, without individually identify ing distances to features of the environment. In some examples, the landability model 1100 may be able to take an image of a delivery' location, e.g., as captured by the system-45- 4910-9932-4237\3Docket Number P312830.W0.01 before or at delivery, via satellite, etc., and make a prediction regarding the most ' laudable" areas for the pay load and / or dependent vehicle. The training may utilize 3D modeling, depth, and other models, but once trained can go directly from image to predictive location, without intermediate depth, mapping, and other steps. This allows a faster compute time and more accurate as the images of a delivery location can be directly used to predict landable areas without separate processing steps. However, such a model may require extensive training data of images and successful landing areas and so may take time to develop and train.
[0224] The compute resources 1020 may associate or combine the determined information to identify features of the delivery' environment, orientations of the aerial vehicle 102, 200 in the environment, model the delivery environment, and / or update or determine a flight path. The generated or determined information may ultimately be used to generate information used by the aerial vehicles 102, 200 to traverse a flight path and / or deliver a payload 101.
[0225] The computer resources 1020 may include an image rectification module 1240. The image rectification module 1240 may be a machine learned model and may be stored on the aerial vehicles 102, 200 or operatively in communication with the aerial vehicles 102, 200 and / or may be stored on one or more remote compute resources. Image rectification 1240 may be a compute resource 1020 that identifies and associates corresponding or matching points between information captured by the sensors 1040. The information captured by the sensors 1040 may vary due to differences in time the information was captured, such as between a first capture and a subsequent capture. The differences in the information may be differences in vantage points, such as between the two or more stereo cameras 244. In some examples, the differences may be between different types or sources of information captured, such as between the monocular camera 242 and the stereo cameras 244. For example, image rectification 1240 may identify corresponding pixels in two or more different images captured by visual sensors 1040 or corresponding distances or features captured by depth sensors 1040. As a result, image rectification 1240 may assist in defining a frame of reference or points of comparison between sets or portions of information captured by the sensors 1040.Accordingly, in some examples, image rectification 1240 may provide or assist in-46- 4910-9932-4237\3Docket Number P312830.W0.01 determining positions of the various objects or the aerial vehicles 102, 200 relative to the delivery environment by commonly identified features in the inputs or images.
[0226] With reference to the images captured by the sensors 1040. such as from the wide angle 242 or stereo pair cameras 244, the images may be captured continuously, after predetermined increments of time, or in single captures (e.g., discretely). However, because the aerial vehicles 102, 200 may be in motion, or vibrations may be experienced due to wind or operation of the propulsion systems 140, 220. each frame or image may be captured from a different perspective or orientation relative to the environment. Accordingly, the shift may create the appearance of movement in a stationary object, the appearance of a different magnitude or direction of movement of a moving object, or difficulty in determining dimensions or locations of the objects. The image rectification 1240 may analyze the images or execute one or more operations to account for the vibrations or movement of the aerial vehicles 102, 200. For example, image rectification 1240 may compare pixels or points in images amongst the variety7of images to identify common points of reference.
[0227] Changes in the points of reference may be compared or tracked over time to determine the changes in orientations resulting from movement or vibrations of the aerial vehicles 102, 200. The images may then be adjusted or arranged by image rectification 1240 to remove the effect of the vibrations (e g., adjust the frame of reference for the images). As a result, the image rectification 1240 may arrange or adjust the images to provide a more accurate or consistent view of the environment as captured by sensors 1040 on a moving vehicle.
[0228] The compute resource 1020 may include, in various examples, a depth estimation model 1160 and / or a semantic understanding model 1180. The depth estimation model 1160 and / or a semantic understanding model 1180 may be machine learned models. The models 1160, 1180 may determine or identify characteristics of the environment in which the aerial vehicles 102, 200 navigate. The depth estimation model 1160 may determine distances between the aerial vehicle 102, 200 and various objects or features of the environment. The semantic understanding model 1180 may identify or classify various objects or features of the delivery7environment, e g., determine that certain images are representative of trees, houses, and other common objects, such as by determining pixel characteristics associated with such objects. The models 1160, 1180-47- 4910-9932-4237\3Docket Number P312830.W0.01 may receive inputs, such as images, directly from the sensors 1040 or from the image rectification module 1240. In various examples, the ML models 1160, 1180 may, collectively or individually, obtain from two-dimensional imagery on an environment information about pixels within the two-dimensional imagery. The information may be in real time as the deliver}' is occurring and / or pre-flight, such as via a deliver}' location validation process that validates a location as being available for deliveries. In many instances, preflight images may be captured and used as training data or to make initial assessments regarding where a payload could be delivered. The system will also use real time images captured during deliver}' attempts to navigate the environment to the delivery location.
[0229] The information may be depth or distance, colors or contrast, or the like. Such information may be used by the aerial vehicle 102, 200 to identify or determine how far away obstacles in the environment are from the aerial vehicle 102, 200 during navigation (e.g., forward flight, descent of the secondary vehicle 200). As discussed herein, the models 1160, 1180 may provide information to use for the generation of a three-dimensional model of the environment, such as by a three-dimensional ("3D”) World Modeling Module 1280. Such a three-dimensional model and / or other depth information may therefore be utilized to navigate the aerial vehicle 102, 200 through an environment. In some examples, the ML models 1160, 1180 may provide information to one another to assist in generating, refining, or improving the information generated from the models.
[0230] FIG. 9A illustrates example inputs 1340 and outputs 1360 from the depth estimation model 1160 to estimate or determine depths or distances between the aerial vehicles 102, 200 and portions of the surrounding environment. As discussed herein, input 1340 to the depth estimation model 1160 may be two-dimensional image data collected using the wide angle camera 242 and / or the stereo camera pair 244 on the aerial vehicle 102, 200. The image data may be an output from the image rectification module 1240. For example, where the sensors 1040 capture images, the depth estimation model 1160 may determine depth or distances of information of pixels relative to the aerial vehicles 102, 200. The image information may be from either aerial vehicle 102, 200 or any of the cameras used by the aerial vehicles 102, 200, such as the monocular 242 or stereo cameras 244. Accordingly, the images of ground based features may often be top-down images or provide a plan view' of the environment.-48- 4910-9932-4237\3Docket Number P312830.W0.01
[0231] The depth estimation model 1160 may generate outputs 1360 which identify distance from the aerial vehicle for the elements in the two-dimensional image. The outputs 1360 may be labels, tags, or updated image data identifying the depth or distance from the aerial vehicles 102, 200. For example, various regions of the image data (e.g., pixels) or depth information (e.g., LIDAR or radar readings) may be categorized or labeled as being at various depths or distances relative to the aerial vehicles 102, 200. In some examples, the depth estimation model 1160 may combine or compare two-dimensional image data of different perspectives at a given location to determine the depths or distances of features. The output from image rectification may assist in identify ing a frame of reference or common points when combining images. For example, the depth estimation model 1160 may compare or combine stereo pairs of images to generate three-dimensional images or data from the combined images to assist in determining a depth or distance. In some examples, the depth estimation model 1160 may be trained to identify the distances of features based on their similarity to sizes of known objects and the appearance of that size at varying distances. For example, if an object or feature is identified having a known size and a known distance, sizes or distances of other objects captured by the sensors 1040 may be inferred.
[0232] The output 1360 depicted in FIG. 9A may be illustrative of a visual output for inspection by an operator. The output 1360 of the stereo depth estimation model 1160 may include associations of distances to various pixels or features of images. In some examples, the outputs 1360 may characterize ranges or discrete distances. Categorizing distances by ranges may reduce the processing power or memory demands of generating outputs 1360. The breadth of the ranges represented by the output 1360 may be varied based on the type of operations or maneuvers of the aerial vehicle 102. 200. For example, such as dunng forward flight along a flight path, broader ranges may be useful to compensate for a greater amount of environment analyzed to reduce memory or processing demands. In some examples, tighter ranges may be utilized when navigation is closer to features (e.g., buildings, payload locations, ground) or during deployment of the secondary vehicle 200 to provide more precise information.
[0233] In some examples, the output 1360 may include or can be represented visually, or a visual model can be separately generated from the output 1360 for inspection by a user or operator. For example, a first distance, or features closer to the aerial vehicle 102,-49- 4910-9932-4237\3Docket Number P312830.W0.01200, may be visually represented by renderings depicting more brightly colored or contrasting features in a visual output 1360 compared to objects at a second or further distance. In some examples, as depicted, closer features may be lighter shades, and more distant features may be darker shades.
[0234] In various examples, the depth estimation model 1160 may be a deep learned model able to infer or receive contextual information regarding the depth information to refine or improve the outputs 1360. The contextual information may be associations between inputs 1340 to the stereo depth estimation model 1160, or associations between information contained within an input 1340, or other additional information relevant to determining the depths or distances. For example, the contextual information may be that a distance associated with a pixel may be similar to pixels having a similar color and in a similar location, or an identification of an object and common sizes or orientations of the object. In some examples, contextual information may be provided by outputs of the semantic understanding model 1180, other compute resources 1020, or additional sensor 1040 information. In some examples, contextual information from the sensors 1040 may include additional image data (e.g., shadows, colors, or patterns across surrounding pixels), the geographic location, orientation, velocity, or elevation of the aerial vehicle 102, 200, such as from the position or environmental sensors 1040. The contextual information may assist in refining or generating more precise outputs 1360. The context may assist in determining or validating the output 1360 from the depth model 1160. For example, map data indicating a road feature or structure may assist in validating a constant or varying depth or distance.
[0235] In some examples, the position of the secondary aerial vehicle 102, 200 relative to the primary aerial vehicle 102 may be used to determine the distance of objects from the primary aerial vehicle 102. For example, the secondary aerial vehicle 200 and the primary aerial vehicle 102 may capture images at similar times and determine when an object is in the proximity of the secondary' aerial vehicle 200. The secondary aerial vehicle 200 may be a known size, or number of pixels, in the images corresponding to different distances from the primary aerial vehicle 102. In some examples, the length of tether 168 deployed may be known and the distance between the secondary vehicle 200 the primary aerial vehicle 102 may be estimated or determined with reference to the length of the tether 168. Accordingly, the information regarding the position of the-50- 4910-9932-4237\3Docket Number P312830.W0.01 secondary aerial vehicle 200 relative to the primary aerial vehicle 102 may be used by the depth estimation model 1160 to improve, refine, or assist in determining a depth or distance of features of the environment.
[0236] The depth estimation model 1160 may be implemented by training a neural network to generate a trained model. The trained model may then be deployed as the depth estimation model 1160 to the aerial vehicle system 100. The trained model 1160 may be deployed to or operatively in communication with the aerial vehicles 102, 200. The neural networks may be trained to recognize or determine distances based on a set of training data including annotations corresponding to known or tagged distances. The depth estimation model 1160 may be trained prior to operation of the system 100 and / or updated during or after operation of the system 100. The depth estimation model 1160 may be trained at a separate or remote computing device, such as a server 118. In some examples, a training network or model may be utilized to assist in training the depth estimation model 1160. For example, the training network may be used to annotate or tag training data, or generate training data. In some examples, the training data may be annotated by a separate operation at another device or by an operator.
[0237] The training data may include information representative of, or similar to, the information captured by the sensors 1040 of the aerial vehicle 102, 200 during use. For example, the training data may include information relating to or representative of depth or distance information. The training data may include or be representative of LIDAR data, captured images, or the like. In some examples, the training data may be previously captured training data of real world environments or synthetic training data including computer generated information representative of real world environments. In some examples, the depth estimation model 1160 may be trained using unsupervised or supervised training methods.
[0238] The captured training data may be images or other information captured by, or similar to information that may be captured by, the sensors 1040. For example, the captured training data may include stereo image pairs as may be captured by the stereo cameras 244 or images similar to those captured by the monocular camera 242. The captured training data may be captured during preceding operations of the aerial vehicle system or by separate aerial vehicles, systems, or sensors. The environments captured by or represented by the training data may be representative or similar to the types of-51- 4910-9932-4237\3Docket Number P312830.W0.01 environments navigated by the aerial vehicles 102, 200. For example, the training data may be of deliver)’ or residential locations, shipping locations, or the like. The training data may include various conditions, such as during the day and night, during precipitation or clear weather, or including various wavelengths of light, such as IR, visible light, or the like. Captured training data may be useful to represent a variety’ of possible combinations of conditions, objects, or the like. For example, no two real world locations are likely to include the same conditions or features. Accordingly, captured training data may provide unique representations of environments to assist a trained model 1160 in recognizing a variety' of complex combinations.
[0239] In some examples, the training data may be synthetic or computer generated training data. Synthetic training data may be simulated data representative of objects or environments existing in the real world. In these examples, the synthetic data may include representative features of an environment (e.g., objects, lighting conditions, weather, or the like). The synthetic training data may include accurate, clearer, or predefined characteristics of represented real world features in a variety of conditions. The synthetic training data may be generated having known or predefined parameters such as varying distances and / or in a variety of lighting or weather conditions. For example, the synthetic training data may include know n or accurate distances to the features, light interactions with objects (e g., IR, visible, or other ranges of the spectrum), physical interactions between representative objects (e.g., wind, movement), and the like. The synthetic training data may provide information or representative conditions missing from the captured training data, difficult to replicate in the captured training data, or for important or likely real world scenarios where additional variations similar to captured training data may be beneficial. It should be noted that the use of synthetic data may generate more accurate models as the “depth’7of the various objects represented in the synthetic images may be completely accurate as compared to real images where the depth may be estimated or limited in accuracy by the sensitivity’ of the real w orld sensor. Thus, by using synthetic data the models can be trained more quickly and be more accurate for at least some conditions.
[0240] In some examples, synthetic training data and captured training data may be used in combination to train the depth estimation model 1160. For example, the synthetic or captured training data may be used similarly as, or supplement, the other set of-52- 4910-9932-4237\3Docket Number P312830.W0.01 training data. By utilizing both captured and synthetic training data, a variety of conditions may be presented to the depth estimation model 1160 during training to assist in preparing the model 1160 for as many possible combinations of conditions as possible.
[0241] The training data may be tagged or annotated. The tags or other identifiers in the training data may be information associated with the various features or objects captured or depicted in the training data. In some examples, the tags may provide context to the training data to assist in training the model 1160 to identify distances or depths of features. The tags may relate or define distances, colors, associations between features, or other information relevant to determining distances or depths of various features. In some examples, the tags may indicate the distances of various features or objects relative to a position or sensors capturing the information. In one example, the tags may be or associated with distances of various pixels in an image.
[0242] In some examples, the synthetic training data may be generated including tags or annotations. Because the synthetic training data may be generated as representative of various real world features or predefined characteristics, predefined parameters (e.g., distances, lighting, environmental factors, types of objects) may be included with the training data as annotations or tags. For example, the synthetic training data may include tags regarding the “depth” of the environmental objects.
[0243] In some examples, the tags or annotations of the training data may be varied or changed. For example, in some sets of training data at least some of the existing tags, annotations, or portions of information from the training data may be used or altered. For example, training data may be used more than once including different sets of tags. The training data may be altered or limited so that during training the model is required to analyze multiple combinations of features.
[0244] Multiple types of training data, or sources of training data, may be combined or used to generate tags. In some examples, the training data from or corresponding to a first sensor 1040 may be tagged or associated with training data corresponding to a second sensor 1040. For example. LIDAR data, or other systems providing true depths or distances information of features, may be captured with the training data. The true depths or distances may be overlaid or combined with a first set of training data to associate the first set of training information, such as images or pixels in the images, with an actual-53- 4910-9932-4237\3Docket Number P312830.W0.01 depth or distance. By comparing the true depth or distance with the training data, the training network may assist in guiding the training of the depth estimation model 1160 to predict actual distances.
[0245] The training data may include information comparable to the outputs of the other compute resources 1020, such as identified features by the semantic model 1180 that may be used by the depth estimation model 1160 during use. In such an example, the tags may be applied to the training data to assist the depth estimation model 1160 in associating various features with changes in depths or distances. The overlaying or combination of sources of information may be represented or stored in a three-dimensional ("3D") point cloud, or association of the various identified features with a vector or point indicating its position in three-dimensional space.
[0246] During training, the depth estimation model 1160 may be presented with the various training data. The depth estimation model 1160 may be trained to determine or predict distances of the presented information. The depth estimation model 1160 may compare the predictions to the known distances or features. Differences in predictions may be corrected and the model 1160 may be updated to improve predictions for similar sets of information. In some examples, the depth estimation model 1160 may be trained to recognize additional or contextual information to assist in determining a depth or distance. For example, tags present in the training data may assist the depth estimation model 1160 to determine or improve determinations of distances based on additionally present characteristics, such as common colors, reflectiveness or light intensity, or other relevant information. The depth estimation model 1160 may be updated or improved over time as the aerial vehicle 102, 200 captures additional information. For example, customers or recipients of deliveries may provide feedback regarding accuracy of delivery location. Such feedback may be provided to the depth estimation model 1160 to increase its accuracy over time.
[0247] Turning to the semantic understanding model 1180, the semantic understanding model 1180 may identify or classify' characteristics about an environment by information captured by the one or more sensors 1040. The semantic understanding model 1180 may, in various examples, be implemented using a classifier, neural network, or other type of machine learned model. FIG. 9B illustrates example inputs 1380 and outputs 1400 from the semantic understanding model 1180. The semantic understanding model 1180 may-54- 4910-9932-4237\3Docket Number P312830.W0.01 identify objects, features, areas or the like from the information received by the sensors 1040 or other inputs to the system 100. For example, the semantic understanding model 1180 may analyze the captured information to identify types of objects, natural elements, people, animals, or the like. Input 1380 to the semantic understanding model 1180 may be two-dimensional image data collected using the cameras of the aerial vehicles 102, 200 or other two-dimensional image data. For example, the semantic understanding model 1180 may receive the two-dimensional image data and / or the estimated depth information from the depth estimation model 1160 to perform semantic segmentation to identify the particular features or objects in the delivery environment in the images. In some examples, the semantic understanding model 1180 may receive outputs from the image rectification module 1240 as an input 1380. In some examples, the input 1380 may include additional information captured by the sensors 1040 including location, orientation, elevation of the aerial vehicles 102, 200, or other information relevant to identification of features of the environment.
[0248] In some examples, information received from a delivery request may be used to provide contextual information to the semantic understanding model 1180. For example, a delivery request may provide a delivery address, specify a preferred payload location, areas to avoid delivering the payload 101, or delivery' instructions. The specified or preferred location may be indicated by an address, a set of coordinates, a labeled map or image, or natural language text. If a location is specified, the compute resources 1020 or system 100 may identify or estimate the location by comparing to existing information, such as a map or images of the location. In examples including natural language text, the system 100 or compute resources 1020 may analyze the text, such as by a machine learned model, to identify information included in the delivery request. The information provided in the delivery request, or other contextual information, may be analyzed or interpreted to identify, or predict, and confirm the presence objects or features in the environment. In such an example, the information of the delivery request may be analyzed or compared to existing information, such as existing maps, satellite imagery, customer provided images, or the like. For example, existing information may indicate or disclose the presence of a tree or structure at the location. The semantic understanding 1180 may use the existing information to predict the presence of or increase a confidence or weighting of an identification of a tree at the location. After determining the contents-55- 4910-9932-4237\3Docket Number P312830.W0.01 of the delivery request, the information may be used or combined with analysis of the information captured by the sensors 1040 to weight the model 1180 to improve predictions made by the model 1180.
[0249] The output 1400 from the semantic understanding model 1180 may be classifications or identifications of various features of the inputs 1380, such as of pixels within the two-dimensional image data. The classifications may be for a single input or image, sets or series of inputs, or the like. For example, the semantic understanding model 1180 may identify objects or characteristics of the delivery environment such as pavement or other solid surfaces, dirt, plants, grassy areas, static items (e.g., lawn furniture, tools, and the like), lines, humans, or the like. In some examples, the output 1400 may group objects by a type or classification rather than individually identify objects. For example, features may be categorized based on whether they should be avoided or whether the feature is a type associated with potential landing or delivery areas.
[0250] To identify objects or characteristics of the delivery environment, the semantic understanding model 1180 may analyze the images for various identifiers of the objects or characteristics of the delivery environment. For example, the various identifiers may include shapes of features, colors or contrast between features, movement, sizes, or various other visual or physical identifiers of features and objects. In some examples, the identifiers may be collectively analyzed and the various features may be identified and associated with objects or features. For example, the semantic understanding model 1180 may be trained to recognize that the presence or combination of the various identifiers associated with various known objects or features. When more of the identifiers associated with an object are identified, the semantic understanding model 1180 may¬ weight or predict a certain output 1400 identifying or predicting one or more features are present in the environment. In some examples, when an unknown feature or unknown combination of features is identified, the model 1180 may predict the type of feature based on a similarity to a known object.
[0251] In some examples, information from the depth estimation model 1160. e.g., output 1360, may assist in contextualizing and identifying objects or features. The depth estimation model 1160 may provide distances or sizes of various features present in the captured information. The distances or sizes may assist in determining or differentiating-56- 4910-9932-4237\3Docket Number P312830.W0.01 between likely features identified by the semantic understanding model 1180. For example, a variation in height or depth from the aerial vehicles 102, 200 of a feature, or lack thereof, may assist in classifying the feature.
[0252] With reference to shapes of features, geometric shapes may be indicative of man-made objects. Colors, or changes in colors, of the various features may be indicative or characteristic of certain features. For example, large green patches may be indicative of grass or turf. Grass or lawn may be further or alternatively classified based on a similarity in height across a region. Various objects may be identified or weighted towards a classification based on the reflective characteristics when exposed to a light source. For example, glass or polished surfaces, such as tables, may be more reflective than their surroundings. Whether the object is stationary or in motion may be determined by analysis of the various inputs 1380 over a period of time. For example, a first image may be compared to a second image captured at a separate time. The location or position of the object may be determined in each image relative to the aerial vehicles 102, 200, or compared between the images to determine if the analyzed feature or object has changed a position. In some examples, patterns of movement may be identified or associated with certain features, such as the steps of humans or animals. Infrared filters on cameras or other sensors 1040, may assist to identify variations in temperature between features. The variations of temperature may distinguish boundaries of objects or identifying characteristics of an object. For example, power lines may have an increased temperature over surroundings. Buildings may be colder or warmer than a surrounding natural environment.
[0253] In some examples, the semantic understanding model 1180 may identify a status or condition corresponding to the classified object or feature. For example, after identifying or classifying a feature, the input 1380 may be further analyzed for related or common features or objects associated with the classified feature. In one example, the semantic understanding model 1180 may initially identify a driveway or sidewalk. The semantic understanding model 1180 may further determine whether the classified sidewalk or drive is cluttered or otherwise occupied, such as by the presence or order of objects in the space, such as cars, people, toys, or the like. If the driveway is clear, it may be identified differently from a parking lot including cars, such as a preferred payload location. In some examples, behaviors or actions of classified features may be known or-57- 4910-9932-4237\3Docket Number P312830.W0.01 predicted and the input 1380 may be analyzed to determine the presence of those actions or behaviors. For example, one or more cars parked neatly may be indicative of a space unoccupied by humans, while cars with doors open may be indicative of an area in use. In another example, the semantic understanding model 1180 may detect that if people are adjacent a table and it includes objects, it may be indicative of a meal or other use. The identification of objects or characteristics of the delivery environment may provide additional context for the aerial vehicle 102, 200 to maneuver during descent and / or to locate delivery areas where a precise delivery location is not identified. For example, the aerial vehicle 102, 200 may prioritize avoidance of certain identified feature or object categories (e.g., people, moving vehicles, trees, wires, etc.).
[0254] In some examples, the semantic understanding model 1180 may include or be models 1180 designed to detect certain features, such as the presence of wires, buildings, areas associated with landing or delivery, humans or animals, roads, or various other features or objects potentially expected in a delivery7environment. For example, the one or more semantic understanding models 1180 may be trained to identify wires. The semantic understanding model 1180 may be trained to identify thin objects and estimate or predict connection points for an object to other objects to assist in validating or identifying the objects or features in the captured information. The wires or other features may be identified in changes in temperature relative to the surroundings, such as by IR light. In some examples, the semantic model 1180 may identify boundaries of the various objects, locations, or features. For example, if a wire is identified, the model 1180 may determine a direction, size, attachment or end points, or related information of the wire.
[0255] The semantic understanding model 1180 may be implemented by training a neural network to generate a trained model. The trained model may then be deployed as the semantic understanding model 1180 to the aerial vehicle system 100. The neural networks or the semantic understanding model 1180 may be trained to identify' or classify' objects or features in information captured by sensors 1040. The semantic understanding model 1180 may be trained on training data including tags or annotations identifying known objects or features. The semantic understanding model 1180 may be trained prior to operation of the system 100 and / or updated during or after operation of the system 100. In some examples, a training network or model may be utilized to assist-58- 4910-9932-4237\3Docket Number P312830.W0.01 in training the semantic understanding model 1180. For example, the training network may be used to annotate or tag training data, or generate training data. In some examples, the training data may be annotated by a separate operation at another device or by an operator.
[0256] The training network may include training data representative or similar to the information captured by the sensors 1040. In some examples, the training data includes information representative of one or more objects or features within or likely to be within a flight pathway or delivery environment. For example, the training data may include audio recordings, images, or the like. The training data may include information captured training data, including information captured in a real world environment, or synthetic or computer generated training data representative of real world environments. In some examples, the semantic understanding model 1180 may be trained using unsupervised or supervised training methods.
[0257] The captured training data may be images or other information captured by, or similar to information that may be captured by, the sensors 1040. For example, the captured training data may include stereo image pairs as may be captured by the stereo cameras 244 or images similar to those captured by the monocular camera 242. The captured training data may be captured during preceding operations of the aerial vehicle system or by separate aerial vehicles, systems, or sensors. For example, the training data may be of regions surrounding or including delivery or residential locations, shipping locations, regions between the two locations, or the like. The training data may include various lighting conditions, such as during the day and night or during precipitation or clear weather, or various wavelengths of light, such as IR, visible, or the like. Captured training data may be useful to represent a variety of possible combinations of conditions, objects, or the like. For example, each real world location likely to includes at least some unique combination of conditions or features. Accordingly, captured training data may provide unique representations of environments to assist in training a model 1180 in recognizing a variety of complex combinations.
[0258] In some examples, the training data may be synthetic or computer generated training data. Synthetic training data may be simulated data representative of objects or environments existing in the real world. In these examples, the synthetic data may include representative features of an environment (e.g., objects, lighting conditions,-59- 4910-9932-4237\3Docket Number P312830.W0.01 weather, or the like). The synthetic training data may include accurate, clearer, or predefined characteristics of represented real world features in a variety of conditions. The synthetic training data may be generated having known or predefined parameters such as objects or features at various orientations or varying distances, and / or in a variety of lighting or weather conditions. For example, the synthetic training data may include known or accurately defined features, light interactions with objects (e.g., IR, visible, or other ranges of the spectrum), physical interactions between representative objects (e.g.. wind, movement), and the like. The synthetic training data may provide information or representative conditions missing from the captured training data, difficult to replicate in the captured training data, or for important or likely real world scenarios where additional variations similar to captured training data may be beneficial. It should be noted that the use of synthetic data may generate more accurate models as the features or characteristics of the various objects represented in the synthetic images may be completely accurate as compared to real images where a complete depiction of the figure may be estimated or limited in accuracy by the sensitivity or effective ranges of a real world sensor. Thus, by using synthetic data the models can be trained more quickly and be more accurate for at least some conditions.
[0259] In some examples, synthetic training data and captured training data may be used in combination to train the semantic understanding model 1180. For example, the synthetic or captured training data may be used similarly as, or supplement, the other set of training data. By utilizing both captured and synthetic training data, a variety of conditions may be presented or analyzed during training to assist in preparing the model 1180 for as many possible combinations of conditions as possible.
[0260] The training data may be tagged or annotated. The tags or other identifiers in the training data may be identifiers of one or more characteristics of the various features or objects and / or identification of those features. For example, the tags may be or associated with the characteristics of various features or the types of features in the training data. The training network may generate tags applied to the training data or the tags may be generated by a user or during a separate operation. In some examples, the tags may provide context to the training data to assist in training the model 1180 to identify objects or features.-60- 4910-9932-4237\3Docket Number P312830.W0.01
[0261] The training data may include information comparable to the outputs of the other compute resources 1020, such as determined distances by the depth estimation model 1160. For example, understanding or associating the sizes of various objects or features at different distances may assist in identifying or differentiating between possible features. The tags may be applied to the training data to assist the semantic understanding model 1180 in associating various features with changes in depths or distances, or varying perspectives. The overlaying or combination of sources of information may be represented or stored in a three-dimensional format, with reference to the depth estimation model 1160, or association of the various identified features with a vector, string, or other formats.
[0262] In some examples, the tags or annotations of the training data may remove existing tags, annotations, or portions of information from the training data. For example, training data may be used more than once including different sets of tags. The training data may be altered or limited so that during training the model is required to analyze multiple combinations of features.
[0263] In some examples, multiple types of training data may be combined or used to generate tags. The combined training data may be or provide contextual information. For example, the training data from or corresponding to a first sensor 1040 may be tagged or associated with training data corresponding to a second sensor 1040. For example, images may be combined with map data to recognize relationships between maps and captured images. The contextual information may be overlaid or combined with the training data to associate the additional information with an identification of a feature or classification. By comparing the tagged information with the captured information, the training network may assist in guiding the training of semantic understanding model 1180 to predict or identify actual features in an environment.
[0264] During training, the training model or network may present semantic understanding model 1180 with the various training data. The semantic understanding model 1180 may be trained to determine or predict features or objects in information captured by the sensors 1040. The semantic understanding model 1180 may compare the predicted features or classifications to the tags indicating one or more characteristics or classifications of the features. Differences in predictions may be corrected and the model 1180 may be updated to improve predictions for similar sets of information.-61- 4910-9932-4237\3Docket Number P312830.W0.01
[0265] In some examples, the semantic understanding model 1180 may be updated or improved over time as the aerial vehicle 102, 200 captures additional information. In some examples, the captured information by the aerial vehicles 102, 200 may be added to the training data or tagged for use in training. For example, the semantic understanding model 1180 may receive two-dimensional image data captured by the aerial vehicle 102. 200 and / or the estimated depth information from the depth estimation model 1160 to perform semantic segmentation to identify the particular features or objects in the delivery environment in the captured images. Semantic segmentation may further identify classes or attributes of the objects in the delivery environment.
[0266] While specific examples are given with respect to separate semantic understanding 1180 and stereo depth 1160 models, the models or functionality of the models can be performed by a single model, such as the landabilify model 1100. In some examples, the separate image rectification model 1240 is omitted and the landabilify model 1100 can identify frames of references or associate features or data between images.
[0267] The landability model 1100 is trained on information to interpret captured data to identify landable zones or potential delivery locations based in part on the types of objects or the elevations or distances between objects. The landabilify model 1100 can be trained on the various possible training sets described herein with reference to the depth estimation model 1160 (distances and elevations) and the semantic understanding model 1180 (e.g. object classifications). Through training on both depth and object classification training data, the landabilify model 1100 is trained to identify patterns in potential payload or landing locations based on object classifications and distances of objects. In identifying the patterns and associations between object classifications and distances, the landable zones can be identified without separately identifying the specific objects or distances. As a result, the processing speed of a single trained landabilify model 1100 may be faster than separate operations of the depth 11 0 and semantic models 1180 and optionally may have a reduced processing load as compared to separate models. Reducing processing demands can be beneficial for aerial vehicles where power reserves and total mass can be constraints on the range of operation of the aerial vehicles 102, 200. In short, the landability model can generate predicted landing locations for both the dependent aerial vehicle 200 and / or the pay load faster than some of the-62- 4910-9932-4237\3Docket Number P312830.W0.01 incremental models as it can take one or more images captured by the system (e.g., by the camera on the dependent vehicle) and translate those directly into a landable area prediction.
[0268] Using a single model 1100 for at least depth estimation and semantic understanding can also be beneficial for integrating feedback into training data to continue to improve the model. For example, two separate models can require training data from a single source to be separately annotated, formatted, or otherwise handled separately into data sets for each model. A single model 1100 can omit duplicative processing, and in some examples can be trained on richer or more complex training sets. In some examples, a single landability model 1100 can utilize training data including information or formatting corresponding to either or both models.
[0269] The landability model 1100, or alternatively the semantic understanding model 1180 separately, may generate a desirability ranking of payload locations based on the objects identified by one or more of distances, semantic segmentation, world modeling or the like. The desirability ranking, or landability ranking, may guide the selection of payload locations for packages by the aerial vehicle 102. 200. The desirability ranking may be based on rules or preferences regarding payload locations. For example, the desirability ranking may include preferences based on geometry, e.g., a flat surface is preferred over a non-flat surface for a payload location. The desirability ranking may include preferences based on classification of semantic objects, e.g., delivery on a concrete surface is preferred over delivery on a grass surface, delivery near a door is preferred over delivery on a bush or tree. In some examples, the landability model 1100 is trained to recognize patterns between features that are desirable for delivery (e.g. the color of cement or grass, the sidewalks leading to a structure indicating a door, or the like) and identify regions having these patterns rather than individually classifying the objects or distances. For example, the landability model 1100, or the semantic understanding model 1180, may extrapolate information based on the observations of an environment to generate the desirability ranking. For example, where a sensor captures information that includes a walkway in the delivery environment, the semantic understanding model 1180 may extrapolate that the region is desirable for a delivery based on patterns in the training data that indicates a sufficient probability a door exists at the end of the walkway even if the door is obscured in the captured image.-63-4910-9932-4237\3Docket Number P312830.W0.01
[0270] The desirability ranking may include exceptions or stipulations for the preferences, or the model 1100 can adjust based on a delivery request (e.g. user feedback, mission parameters, observed conditions, or the like). For example, the landability model 1100, and in another example the semantic understanding model 1180, may select or identify a less preferred payload location over a more preferred payload location if the less preferred payload location would avoid a turbulent descent during delivery. The compute resources 1020 may update the desirability ranking based on collected data from package deliveries, missions, user feedback, or the like. For example, where a package delivery failed at a highly preferred payload location class, the semantic understanding model 1180 may further analyze the environment to confirm or update classifications of objects and in turn the desirability ranking to lower the desirability of the payload location class.
[0271] In some examples, customers or recipients of deliveries may provide feedback regarding perceived navigation or interpretations of the aerial vehicles 102, 200 or feedback regarding delivery preferences. For example, customer feedback may include an indication that the customer preferred or did not prefer the payload location chosen for the package. In such examples, the feedback can be added to the training data or interpreted by the landability model 1100 or semantic understanding model 1180. The models can be retrained or adjust a weighting based on a natural language training to associate feedback in the language with observed characteristics or outputs at the payload location. In turn, the training or analysis may update the desirability ranking based on the customer feedback, such that positive feedback for a payload location increases the desirability ranking of such a pay load location or pattern of features at the pay load location. By such means, the ML models forming the compute resources 1020 may iteratively leam to improve payload location selection in a delivery environment.
[0272] In some examples, the compute resources 1020 can analyze all or only a portion of a delivery' environment. The compute resources 1020 may limit analysis of an environment to finding a first delivery' location, a region of space needed for delivery (e.g. deployment of the dependent vehicle 200), or based on predefined preferences for a location. In one example, the semantic understanding model 1180 and / or the depth estimation model 1160 identify all features included in information from the sensors 1040, or only a portion of the information. With respect to an image, the entire image or-64- 4910-9932-4237\3Docket Number P312830.W0.01 a portion of the image may be analyzed. With respect to contents of an environment, a predetermined set or certain objects or characteristics of the delivery environment may be first analyzed or identified.
[0273] Analyzing only a portion of captured information, such as an image, or analyzing an image for a subset of objects or characteristics, may reduce the processing or memory' requirements of the system 100 or compute resources 1020. For example, limiting the intended outputs from, or a total amount of inputs to, the models 1100, 1240, 1160, 1180 may result in quicker or less demanding generation of the outputs. For example, with respect to a delivery of a payload 101, if a delivery request specifies a preferred payload location, the models 1100, 1160, 1180 or compute resources 1020 may determine information about, attempt to identify, or validate information about only that payload location. In some examples, certain objects may be determined as more important to avoid, such as animals or humans, or wires. Other objects may either not be analyzed or identified individually. In some examples, certain objects or characteristics may only be analyzed if other conditions are detected. For example, a second payload location may only be analyzed if a first payload location does not meet landabilify criteria described herein.
[0274] In some examples, a location may be associated with information prior to arrival by an aerial vehicle 102, 200 and known to the system 100. For example, the location may have been part of the training data sets, previously visited by another aerial vehicle 102, 200, or otherwise included in information used by the system 100. In some examples, the location may have been analyzed, whether autonomously or by an operator, and objects or flight corridors previously identified. For such locations, the system 100 may store or generate predefined flight paths that navigate about the features or through the environment. The system 100 may pre-generate or store outputs from the compute resources 1020, such as identification of objects or distances. In some examples, at the known location the various objects or features may be masked, such as by the compute resources, or operation of the sensors 1040 may be reduced or limited. The generated outputs may then be accessed or communicated to an aerial vehicle 102, 200 associated with the location to reduce processing or delivery times. In some examples, the masking or pre-generation of flight paths may result in more precise navigation, such as to avoid obstacles or operate at quicker speeds. As a result,-65- 4910-9932-4237\3Docket Number P312830.W0.01 navigation of the aerial vehicles 102, 200 at the known locations may be more accurate, quicker, or require fewer processing operations.
[0275] The compute resources 1020 may utilize the image data and / or outputs 1360, 1400 to generate depth information and classifications for the pixels in the two-dimensional image data. In some examples, the landability model 1100 models an environment without separately identifying specific distances or types of objects.Alternatively, such depth information and classifications may be provided to a three-dimensional world modeling module 1280, which may generate a three-dimensional model of the environment being navigated. Accordingly, features and functionality described with reference to the landability model 1100 or the three dimensional world model 1280 can alternatively be performed by either model separately, or the features of three-dimensional world modeling module 1280 can be performed by the landability model 1100.
[0276] The three-dimensional model generated by a landability model 1100 or three-dimensional world modeling module 1280 may, in various examples, be a three-dimensional point cloud or information defining depths or distances to various objects and estimations of their sizes relative to the aerial vehicles, such as a listing of coordinates or three-dimensional reconstruction. The model may be representative of an environment of the aerial vehicles 102, 200. In other examples, the ML models may identify areas where objects are not present to identify potential flight corridors. In such an example, the ML model (e.g. landability model 1100) may interpret capture data to identify open regions of airspace having sufficient volume to navigate an aerial vehicle 102, 200.
[0277] Three-dimensional world modeling 1280 may additionally or alternatively utilize information received from image rectification 1240. navigation 1220. and / or other components of the compute resources 1020, as discussed herein, to generate the three-dimensional model 1280. The three-dimensional world modeling module may be a machine learned model or executable operation, such as by one or more processing elements.
[0278] With reference to FIG. 10, graphical examples of operation of the three-dimensional world modeling module 1280 are depicted. The graphical example is intended to illustrate potential operations of the models and a visual output may not be-66- 4910-9932-4237\3Docket Number P312830.W0.01 reproduced in operation. A three-dimensional model 1500 may be generated by three-dimensional world modeling 1280 based on outputs 1400 from the semantic understanding model 1180 and / or the depth estimation model 1160. In other examples, the landability model 1100 can generate a model based on similarities in patterns to training data including depth and semantic information in three dimensional environments. The landability 1100 or three-dimensional world modeling module 1280 may associate identified objects and distances, or identify patterns between objects at various distances, such as from the models 1160, 1180, to 3 dimensional coordinates to generate the three dimensional model 1500.
[0279] In some examples, the location or position of the aerial vehicles 102. 200 may¬ be determined with respect to a generated three dimensional model 1500, or the model 1500 can be generated in part on the known position of the aerial vehicles 102, 200 and determined points or distances relative to the aerial vehicles 102, 200. The three-dimensional model 1500 may assist in locating or determining the presence of preferred or navigable landing spots or flight paths, such as by identifying volumes spaced from or between two obstacles (e.g.. vegetation, structures, or other obstacles). In various examples, three-dimensional world modeling 1280 may identify areas within the three-dimensional model 1500 as preferred delivery locations, such as those areas marked without obstacles or bounded by other indicia in the three-dimensional model 1500.
[0280] In various examples, three-dimensional world modeling 1280, or the landability model 1100, may utilize or generate a three-dimensional occupancy grid 1502 to determine or identify navigable spaces in the three-dimensional model 1500. The occupancy grid may be visually represented as an environment where each grid cell (or voxel) is provided with a value reflecting a probability that the voxel is free or occupied. The three-dimensional occupancy grid 1502 may, in operation, be executed or stored as strings, matrixes, or various other combinations of value to represent the positions and classifications of the voxels. In some examples, if a threshold probability- is met, the voxel 1506 may be represented as occupied, or a voxel 1504 may be indicated as unoccupied. The occupancy grid 1502 may be visually represented by values, tags, or colors to represent that a voxel 1506 includes an obstacle or is proximate to an obstacle to be avoided by the aerial vehicle 102, 200, that the voxel 1504 is open space or navigable, or that the voxel 1510 is unobserved or has an unknow n status. For example,-67- 4910-9932-4237\3Docket Number P312830.W0.01 in the three-dimensional occupancy grid 1502, voxels 1504 shown in white are free. In some examples, voxels 1508 that are not occupied, but are close to an object may be identified as occupied or as voxels 1508 in close proximity to an object. For example, a moving object may be represented as an occupied voxel 1506 and possible changes in position may be represented by the proximal voxels 1508. These proximal voxels 1508 may be labeled or associated with a third identifier to identify regions to avoid if possible or open to navigation if alternative routes change in availability. During operation, the occupancy grid 1502 may be continuously, periodically, or discretely updated after a duration of time or a distance traveled by the aerial vehicles 102, 200. For example, as additional information is received by the sensors 1040, the three-dimensional model 1500 may be updated.
[0281] With reference to the operation of the landability model 1100, or separate operations of the semantic understanding model 1180 or the depth estimation model 1160, the voxels may be used to summarize regions of space to reduce processing demands for characterizing the outputs of the models 1160, 1180. For example, while precise distances may be determined by the depth estimation model 1160, the height or width of the voxels may summarize ranges of distances to reduce processing demands for determining flight paths or other operations. For the semantic understanding model 1180, the identified objects may be summarized by voxels indicating their location. Similarly, the landability model 1100 can generate a model identifying regions as open for navigation, an object, or a potential landing area among other potential classifications. By representing the objects as occupied space, rather than discrete objects, the memory or processing demands may be reduced. For example, the size or number of the voxels 1506, 1508 may correlate to a size of the object or feature represented, a tolerance for interaction between the aerial vehicles 102, 200 and features of the environment, maneuverability of the aerial vehicles 102, 200, or the like. A moving or oscillating object may be represented by one or more voxels 1508 corresponding to a greater volume or size than the actual object. In some examples, known or predicted characteristics of identified objects or features, such as by the semantic understanding model 1180, may be used to determine a size or number of voxels. For example, a car may include voxels where a door may be opened or where movement or changes in volume may be anticipated. In some examples, objects-68- 4910-9932-4237\3Docket Number P312830.W0.01 identified as humans or animals may receive an additional, unique pattern, or larger voxel to prevent an aerial vehicle 102, 200 operating within a certain proximity to the object.
[0282] With continued reference to Fig. 8, description is provided with reference to models or compute resources 1020 for navigating the aerial vehicle at least partially autonomously through an environment based at least in part on the interpretation of captured information. While description is given to individual compute resources 1020 providing defined functions, it is appreciated the functions or outputs of the modules 1020 can be consolidated to a single model or resource or distributed across two or more separate model or modules. For example, the navigation 1220, localization 1260, mission planner 1320, or trajectory planning 1300 models can be integrated in a larger model, such as the landability model 1100.
[0283] During navigation of the aerial vehicle 102, 200, the sensors 1040 or information derived from captured data by the compute resources 1020 may provide input to the navigation model 1220. The navigation model 1220 may utilize information from the sensors 1040 to determine, predict, or estimate the location of the aerial vehicles 102, 200. For example, navigation 1220 may determine an estimated or initial satellite determined (e.g., GPS) location, such as coordinates, of the aerial vehicle 102, 200, or other positional sensors. Navigation 1220 may track changes in position over time to estimate velocities, accelerations, or directions of travel of the aerial vehicles 102, 200. Navigation 1220 may access or receive map information to determine locations or assist in positioning the aerial vehicles 102, 200. Navigation 1220 may be used to determine relative proximity of the aerial vehicles 102, 200, for example, a distance between the primary aerial vehicle 102 and a deployed secondary aerial vehicle 200. For example, each aerial vehicle 102. 200 may include separate positional or GPS sensors and the differences in coordinates may be compared. In some examples, navigation 1220 may receive information from other components or sensors 1040 of the aerial vehicles 102, 200 associated with the providing thrust or movement of the aerial vehicle 102. 200. For example, the sensors 1040 or components may generate or communicate information indicating that thrust was initiated in a known direction, or control surfaces were actuated to cause a change in orientation in a know n direction. Navigation 1220 may use-69- 4910-9932-4237\3Docket Number P312830.W0.01 the information to predict or identify a change in position, movement, or orientation of the aerial vehicles 102, 200.
[0284] The compute resources 1020 may include a localization module 1260. The localization module 1260 may receive output from navigation 1220 and information from or corresponding to other computer resources 1020 or the sensors 1040 to determine a more accurate or improved understanding of the position or movement of the aerial vehicle 102. 200. In some examples, localization may receive outputs from the landability model 1100, or alternatively either or both of the depth estimation model 1160 or semantic understanding model 1180, the sensors 1040, and / or image rectification 1240. For example, localization 1260 may improve estimated satellite (e g., GPS) coordinates of the aerial vehicles 102. 200 based on the data from the sensors 1040 or other compute resources 1020.
[0285] In some examples, the localization 1260 may determine the location of the secondary aerial vehicle 200 after deployment from the primary vehicle 102. For example, information from the sensors 1040 (e.g., GNSS, tension on the tether 168 and / or the cameras) may be used to determine how the secondary aerial vehicle 200 has moved during deployment from the primary vehicle 102. Such motion information may be used to create an updated estimation of the satellite determined or GPS location of the aerial vehicles 102, 200.
[0286] Some sensors 1040 providing information to the compute resources 1020 may have tolerances or ranges of the accuracy for the identified or measured information, such as determined coordinates by a GPS sensor. The localization module 1260 may use information from additional sensors 1040 of the aerial vehicle 102, 200, or outputs from elements of the compute resources 1020 to reduce or correct tolerances in the accuracy ranges, or to validate the determined information. The localization model may use the information from other sensors 1040 or compute modules 1020 to predict or determine a more accurate location determination to assist in delivery' of a payload 101, or traversal of a flight path. For example, sensors 1040 or compute resources 1020, such as the three-dimensional world modeling 1280, semantic understanding 1180, or depth estimation model 1160, may provide information to localization 1260 to assist in validating or refining the measured information. For example, the semantic understanding model 1180 may identify a street or the depth estimation model 1160 may provide widths or-70- 4910-9932-4237\3Docket Number P312830.W0.01 distances between features of the street, such as houses. The localization model 1260 may compare the identified street or distances to known streets or map information, or measure information or inputs from navigation 1220 to determine a position of the aerial vehicles 102, 200.
[0287] The mission planner module 1320 of the compute system 1020 may include information to identify goals or constraints of the aerial vehicle system 100. For example, the mission planner 1320 may include information about one or more possible or requested flight paths. For example, the mission planner module 1320 may be in communication with the fleet management system 114 or the flight controller 116 and receive or communicate a requested or intended flight path between two or more locations. The mission planner module 1320 may include information about various locations or associated features with the locations such as pickup, delivery, or return locations, information about the payload 101, or various information about the delivery environment (e.g., weather, time, or the like). In some examples, the mission planner 1320 may include information about the capabilities of the aerial vehicles 102. 200, such as possible maneuvers, battery levels, or the like. In some examples, the mission planner 1320 may determine or include an initial flight path for the aerial vehicle 102, 200 between two or more locations. The mission planner 1320 may be in communication with the localization module 1260 to monitor compliance or progress of the aerial vehicles 102, 200 along a flight path.
[0288] The compute resources 1020 may include a trajectory planning module 1300. The trajectory planning module 1300 may determine or refine the flight paths to navigate aerial vehicles 102, 200 through an environment. The trajectory planning model may update portions or entire new flight paths from an initial flight path, or generate corrections based on actual movements of the aerial vehicles 102, 200 to conform to predetermined routes. Trajectory planning 1300 may receive information from the sensors 1040 or one or more compute resources 1020 to generate or update the flight paths. Trajectory planning 1300 may, for example, modify or replace flight paths created by a mission planner 1320 using information from the compute resources 1020 and sensors 1040 to avoid obstacles, navigate around obstacles, or determine a route to a particular delivery’ location, or the like. In some examples, the mission planner 1320 may specify start, end points, or other specific locations as constraints for a flight path and the-71- 4910-9932-4237\3Docket Number P312830.W0.01 trajectory planning module 1300 may generate flight paths between or along the points. In some examples, both the information generated by localization 1260 and the three-dimensional model generated by three-dimensional world modeling 1280 may be provided to trajectory planning 1300, which may utilize such input, along with input from the mission planner 1320. For example, trajectory' planning 1300 may utilize the three-dimensional model 1500 when planning flight paths for the aerial vehicle 102, 200, such as by querying three-dimensional world modeling 1280 for depths or distances between features, or regions of open space within a model or relative to the aerial vehicles 102, 200. In some examples, the trajectory7planning 1300 may be a machine learned model trained to generate one or more flight paths based on the received information.
[0289] In some examples, the trajectory planning 1300 may include predetermined or generated predetermined flight paths. The predetermined flight paths may include information identifying objects or routes to avoid the objects. The flight paths may be between two or more locations, or a portion of the flight path and a start or end point. For example, the predetermined flight paths may define routes to or from a position at the location and a flight corridor for flight over a distance. In some examples, the predetermined flight paths may define routes for the primary7aerial vehicle 102 or the secondary aerial vehicle 200, such as to a delivery7location and / or to a payload location. The predetermined flight paths may be stored on the aerial vehicles 102, 200 or remotely communicated to the aerial vehicles 102, 200.
[0290] In some examples, the flight paths may be to or from for various known locations. For example, the known locations may be previously traversed by one or more aerial vehicles 102, 200, or otherwise previously identified by the system 100, such as in a delivery request. For example, the predetermined flight paths may include information from previous flights to a location, training data, or inputs by operators or users for a location. In one example, the predetermined flight paths may correspond to a shipper, an aerial vehicle docking location, or other commonly visited locations. In another example, the predetermined flight paths may correspond to a delivery location where delivery of a payload was completed successfully one or more times.
[0291] In some examples, the known location may include one or more devices or passively identifying features or symbols configured to identify7the known location to-72- 4910-9932-4237\3Docket Number P312830.W0.01 the aerial vehicle 102, 200. For example, a docking structure may be associated with a marker or fiducial. The fiducials may emit or convey information identifying the known location, structures at the location, or flight path information. In one example, the fiducials may be or include an illuminated display or a predefined pattern that may be detected by the sensors 1040 of the aerial vehicles 102, 200. In some examples, the fiducials may additionally or alternatively operatively communicate with the aerial vehicles 102, 200 or generate signals that may be received by the communication systems 180, 250 of the aerial vehicles 102, 200. The fiducials may communicate by one or more of visible or light signals, audio, electromagnetic emissions or radio frequency, or the various other modalities.
[0292] During operation 1040, the sensors 1040 may be used to identify the fiducials or associated structures. For example, cameras 1040 may capture images of the fiducials. The computer resources 1020, such as the semantic understanding model 1180, may identify the fiducials from captured images.
[0293] In some examples, the fiducials or structures may be associated with predetermined flight paths to. from, or about the structures. Upon identification of the known location or communication with the signal, the trajectory planning module 1300 may retrieve or identify a predetermined flight path. In some examples, the devices may transmit the predetermined flight paths or otherwise identify a predetermined flight path. The predetermined flight path or other information may be communicated directly to the aerial vehicles 102, 200 from the device, such as at the communication systems 180, 250, or may be captured and determined by the sensors 1040. In one example, multiple fiducials may be used to convey information to the aerial vehicles 102, 200. For example, a first fiducial or device may generate, communicate, or correspond to information for a first portion of the predetermined flight path and a second or additional device or fiducial may generate, communicate, or correspond to information for a second portion of the predetermined flight path. For example, the first portion may provide instructions for the aerial vehicle 102. 200 to move or orient itself over a first distance, and the second portion may provide instructions for the aerial vehicle 102. 200 to a second distance from the first distance.
[0294] The use of sensors 1040 to identify the fiducials, docking structure, or objects may provide improved navigation over reliance on satellite determined locations alone.-73- 4910-9932-4237\3Docket Number P312830.W0.01For example, satellite or GPS locations may have a tolerance or common error. As a result, at locations in including multiple landing locations, docking structures, or the like, the errors may prevent accurate navigation to each of the separate structures or locations. By using the sensors 1040, the individual structures may be identified and an improved location may be determined. Additionally, relying on the sensors 1040 and features of the aerial vehicles 102, 200 rather than relying on commands from a separate location may assist in navigating in a variety of situations. For example, the aerial vehicles 102, 200 may navigate during a network outage, power outage, or where an actively communicating external system may be degraded.
[0295] In some examples, the predetermined flight paths may reduce the need to gather or analyze information at the known locations, such as by the sensors 1040. For example, by providing predetermined flight paths, the objects or characteristics of the location may be effectively masked or ignored. By reducing the need to gather or analyze information, complex maneuvers may be used in congested areas, or flight and delivery times may be reduced by quicker operation of the aerial vehicles 102, 200.
[0296] As described with reference to the landability model 1100. in various examples, the computer resources 1020 may identify factors or patterns relating to landability criteria or determine a landability assessment for potential landing locations based on the fitness of a particular area for landing and / or for depositing a payload 101. For example, a landability assessment by the compute resources 1020 may determine whether a payload location satisfies sufficient landability criteria. Landability criteria may be qualifications for determining the viability a location or region, or differences between two or more viable locations, as a payload delivery7location as a landing area or pay load location for the aerial vehicles 102, 200 or location to place a pay load 101. The landability cnteria may be determined or identified based on the presence of features and objects or weighting of the features and objects on their viability The landability criteria may be weighted towards determining a location suited for one or both of the pay load or the aerial vehicles 102, 200. For example, the aerial vehicles 102, 200 may hover or otherwise deploy the payload 101 above a landing surface. In some examples, the type or pay load or characteristics of the payload may require the presence, or lack thereof, of various characteristics at a delivery location that may be similar to or different from the aerial vehicles 102, 200. The landability criteria may be determined by one or more of-74- 4910-9932-4237\3Docket Number P312830.W0.01 the landability model 1100, depth estimation models 1160, semantic understanding model 1180. trajectory planning 1300, mission planning 1320, or the other compute resources 1020 at least in part, or in combination with the other compute resources 1020.
[0297] The landability criteria may be determined, or include factors or weights, based on the presence of various objects or features. A landability model 1100 can recognize patterns between observed objects, distances, and other data relevant to identifying landable zones. In some examples, the semantic understanding model 1180 or depth estimation model 1160 may assist in identifying objects or features relevant to determining landability criteria. For example, the semantic understanding model 1180 may identify the one or more features or regions in a deliver}' environment. Individual features may be identified, or groups of features associated with higher or lower probability- of satisfying landability criteria may be identified, or the regions of space having a combination of relevant features to deliver a payload. In some examples, the three-dimensional world modeling module 1280 may assist or determine a portion of the landability criteria. For example, the model 1500 may be used to determine locations of objects or features, or open areas for the aerial vehicles 102, 200 to traverse. In some examples, the trajectory planning module 1300 or mission planner 1320 may be used. For example, the modules 1300, 1320 may generate flight paths, or determine whether flight paths can be generated, to identified locations from a position of the aerial vehicles 102, 200.
[0298] In determining landability criteria, various objects, features, locations, or areas may be preferred or targeted, such as flat or open areas. In some examples, a landability assessment by the compute resources 1020 may include comparing various regions of the delivery location satisfying the landability criteria. For example, the delivery’ request may define the delivery location or give various locations a preference or levels of priority. The compute resources 1020 (e g. the landability model 1100) can be trained to interpret natural language text to improve or refine outputs. For example, if delivery’ is possible within a distance of a marked location is preferred by a user, the system 100 or compute resources 1020 may attempt to first determine if landing is possible at the marked location or determine the landability criteria of alternative locations, at least in part, based on a proximity to the marked location.-75- 4910-9932-4237\3Docket Number P312830.W0.01
[0299] The landability criteria may be expressed, for example, as a numerical score, a set of values, or the like. In some examples, an optimization algorithm or similar may be utilized to select a landing location from several landing locations based on the landability criteria. In some examples, if a location is identified that satisfies the landability criteria, the determination of landabili ty criteria may cease or continue to identify additional or alternative locations satisfying the landability criteria.
[0300] If multiple locations are identified, the multiple locations may be compared based on the number of features present satisfying the landability criteria or the presence of features weighted as more important or preferred criteria. In some examples, the various objects or features or objects may be features to avoid such as obstacles or objects identified or anticipated to be near the location (e.g., fences, power lines, people, and the like). For example, if an object or feature is identified the landability criteria may be determined based on whether the feature is known to or tagged as likely to interfere with the navigation of the aerial vehicles 102, 200 or delivery' of the payload 101. If the objects or features are identified by the compute resources 1020 having or contributing to a low likelihood of satisfying landability criteria, the compute resources 1020 may proceed to analyze, identify, or determine another location or set of information captured by the sensors 1040 to identify potential other areas satisfying the landability criteria.
[0301] The landability assessment may be based in part on the location of any identified objects or features, such as in the air or on the ground. For example, objects in the air around the location may impede or prevent the secondary aerial vehicle 102, 200 from navigating to a landing. The landability criteria may be based on or include factors or weights relating to the flatness or elevation changes of the location. For example, while the secondary aerial vehicle 200 may deploy a payload 101 without landing, landing on an even surface may be preferred as a more stable landing location.
[0302] The landability criteria may be based on the type of payload, size of payload 101, or packaging of the pay load 101. For example, if the pay load is known to be fragile or round, an area or object having a flat surface may be preferred. In some examples, the surface type of the location or feature (e.g., grass, dirt, concrete, smooth table, and the like) may be considered as landability criteria. For example, if a payload is food, delivery to a dry or clean location may be required. It is appreciated additional factors or-76- 4910-9932-4237\3Docket Number P312830.W0.01 considerations may be relevant or useful in determining landability criteria as understood by one of ordinary skill in the art.
[0303] By using the compute models 1020 to identity a payload location by analyzing naturally occurring environmental objects, or objects having a purpose primarily unrelated to the aerial vehicle system 100, the aerial vehicles 102, 200 may be deployed to numerous or unique environments. For example, by determining landability criteria based on environmental objects, a new delivery recipient may utilize the aerial vehicle system 100 without first acquiring additional equipment or a survey of a desired delivery area.
[0304] The outputs from the computer models 1020, such as trajectory planning 1300, may provide instructions to guidance, navigation and control (GNC) systems 1200 of the aerial vehicle 102, 200. The GNC systems 1200 may include the propulsion systems 140, 220, retraction assembly 160, payload release assembly 230 and / or related components described herein. The computing models 1020 may generate commands for the operation of the GNC systems 1200. For example, the computing models 1020 may include a series of commands, or be in operative communication to relay commands, for the propulsions systems of the aerial vehicles 102, 200 to generate thrust for navigation. The compute resources 1020, such as the trajectory planning 1300 or mission planner 1320, may include or have access to performance information of the various GNC sy stems 1200. The commands or instructions of the compute resources 1020 may be generated by comparing the known performance characteristics to the predicted operational needs based on the determined features of the environment. The instructions to the GNC system 1200 may cause the aerial vehicle 102, 200 to navigate along a flight path, such as to the delivery location, in accordance with the directions and information generated by the compute resources 1020.
[0305] The various compute resources 1020 may be implemented or trained using neural networks. The neural networks may have been trained prior to operation of the system 100 and / or updated during operation of the system 100. For example, the three-dimensional world modeling module 1280, the localization module 1260, or the trajectory planning module 1300 may be implemented or trained by neural networks. The neural networks may be located on the aerial vehicles 102, 200 or at another computing device in communication with the aerial vehicles 102, 200, such as a server 118.-77- 4910-9932-4237\3Docket Number P312830.W0.01
[0306] During training the compute resources 1020 may compare and develop associations between aspects of training data, such as LIDAR information or training images, to assist in modeling an environment or generating outputs. For example, the sensors 1040 may, in some examples, collect data used to train models at another computing device. The compute resources 1020 may be updated or improved over time as the aerial vehicle 102, 200 makes more deliveries utilizing the compute resources 1020. For example, customers or recipients of deliveries may provide feedback regarding accuracy of delivery location.
[0307] FIG. 11 illustrates an example method 2000 of navigating an aerial vehicle 102, 200 in an environment using depth information for the environment. The navigation of the aerial vehicle 102, 200 may be along a flight path between two locations or to a location. The aerial vehicles 102, 200 may carry a payload 101 to or receive a payload 101 at the location. In some examples, the navigation may be by the secondary vehicle 200 from the primary aerial vehicle 102. During navigation, the aerial vehicles 102, 200 may capture information using one or more sensors 1040.
[0308] At block 2020, two-dimensional image data of an environment is gathered using one or more sensors 1040 of the aerial vehicle 102, 200. Two-dimensional image data may be gathered, for example, by the cameras of either or both aerial vehicles 102, 200. For example, the monocular camera 242 and / or the stereo camera pair 244, or the cameras 174. In some examples, other information may be collected in addition to the two-dimensional image data. For example, data from other sensors 1040, such as GNSS and / or IMU information, may help to provide additional context about the location of the aerial vehicle 102, 200 in the environment. In some examples, additional information may be provided by depth sensors (e.g.. radar, LIDAR, additional cameras) such as relative distances between the aerial vehicle 102. 200 and the surrounding environment. Information may be provided by or included in a deliver}' request, such as from a shipper or payload 101 recipient, describing one or more of a preferred deliver}' or landing location, features at the environment, or the like.
[0309] Depth information is obtained from or for pixels in the two-dimensional image data at block 2040. The depth information may be obtained by analyzing the two-dimensional image data by one or more of the compute resources 1020. In some examples, the two-dimensional image data may be analyzed, such as by the image-78- 4910-9932-4237\3Docket Number P312830.W0.01 rectification model 1240, to determine a frame of reference or locations of features relative to the aerial vehicles 102, 200 to compare multiple images. The two-dimensional image data may be an input to the landability model 1100, depth estimation model 1160, and / or the semantic understanding model 1180. In one example, the image data is analyzed for patterns indicating or identifying distances to features in objects. The distances can be a discrete set or range of distances. In another example, the depth estimation model 1160 may estimate depth (e.g., distance from the aerial vehicle 102, 200) for each pixel, or groupings of pixels, in the two-dimensional image data. The semantic understanding model 1180 may utilize the two-dimensional image data and / or the depth estimations to categorize pixels or groups of pixels within the two-dimensional image data. For example, the semantic understanding model 1180 may identify or classify one or more features or objects such as plants, grassy areas, static objects, humans, vehicles, power lines, fence lines, and the like. In some examples, the information generated by the compute resources 1020 may be combined or analyzed in combination with each other. For example, the landability model 1100 can combine object classification with depth information to predict models of the environment without individually analyzing pixels or distances for each object. For example, once a portion of an object is identified at a certain distance, distances and boundaries for the rest of the object can be predicted without or with limited analysis of the captured information. In another example, the three-dimensional world modeling module 1280 can build a model representative of the environment navigated by the aerial vehicle 102, 200.
[0310] At block 2060, the aerial vehicle 102, 200 navigates to a deliver}' location using at least the depth information. For example, the aerial vehicle 102, 200 may navigate along a flight path from a first location to the delivery location. In some examples, the aerial vehicle 102, 200 may navigate from a deliver}- location.
[0311] The three-dimensional world modeling 1280 and / or trajectory planning 1300 compute resources may utilize output from the models 1160, 1180, including the depth estimation information. The compute resources 1020 may identify areas open to navigation or payload locations to place or receive a payload 101. In other examples, the landabilify model 1100 can model the environment directly from the image data.
[0312] The compute resources 1020 may analyze such features to determine maneuvers of the aerial vehicle 102, 200 given the depth estimation information-79- 4910-9932-4237\3Docket Number P312830.W0.01 generated by the depth estimation model 1160 and / or the segmentation generated by the semantic understanding model 1180. Trajectory’ planning 1300 may, for example, modify or generate flight paths created by a mission planner 1320 using the output from the compute resources 1020 to avoid obstacles, navigate around obstacles, locate a particular delivery’ location, or the like. Trajectory’ planning 1300 may generate or communicate commands to control components of the aerial vehicle 102. 200, such as GNC 1200. to maneuver the aerial vehicle 102, 200 along the flight path to the delivery location.
[0313] In some examples, at block 2060, three-dimensional world modeling 1280 and / or trajectory’ planning 1300 may utilize output from one or more of the compute resources 1020, such as depth estimation 1160 information and semantic understanding 1180 to generate landability criteria. The landability criteria may be requirements, preferences, or weights of various factors corresponding to the viability of a potential delivery or payload location. In another example, the landability model 1100 is trained to identify’ or predict areas that satisfy landability criteria without separately weighting or analyzing various factors or weights. Such landability criteria may be utilized to, for example, determine a precise delivery location and navigate to the precise delivery’ location. In some examples, landability criteria may be generated for a number of possible landing locations in a given area (e.g., within property’ boundaries, within a particular radius of a selected or planned payload or landing location, or the like).
[0314] FIG. 12 illustrates an example method 3000 of training neural networks for navigation of an aerial vehicle 102, 200. For example, the method 3000 may be utilized to train the depth estimation model 1160 and / or the semantic understanding model 1180. In another example, the method 3000 can be utilized to train, in whole or in part, the landability model 1100.
[0315] At block 3020, training data is generated for vanous environments. The training data may include captured training data. The captured training data may’ be information similar to or captured by the sensors 1040. In some examples, the training data may be synthetically or computer generated and representative of information captured by the sensors 1040. The captured or synthetic training data may be captured in the same or similar environments associated with various delivery environments.
[0316] The captured training data may be captured by aerial vehicles 102, 200 during previous flights, captured by a specialized aerial vehicle, or include information captured-80- 4910-9932-4237\3Docket Number P312830.W0.01 by potential delivery' recipients. Accordingly, the captured training data may be captured by components similar to or the same as the sensors 1040. The captured training data may include images, audio recordings, depth or distance measurements, or other readings from various sensors. For example, the captured training data may include two-dimensional image data collected by or representative of the images captured by the cameras of the system 100 such as images from a single camera 242. or stereo images captured by stereo pairs of cameras 244, or the like. The captured training data may provide combinations of objects or features in a variety of environments useful to train the models.
[0317] For example, training data may be captured during a previous flight of an aerial vehicle 102, 200. The previous flight by an aerial vehicle 102, 200 may be a piloted or prior autonomous flight to survey data of an environment, and sensors 1040 may capture image frames of the environment, velocity data, and / or location data corresponding to the captured image frames. In some examples, the environment may include planned payload or landing locations.
[0318] The synthetic training data may be generated by a second machine learned model, operation, or training network. The synthetic training data may be similar to captured training data and include or resemble images of an environment, audio recordings, depth or distance measurements, or the like. The synthetic training data may be generated in accordance or including predefined or known parameters for an environment, such as depth or distances to various features, types of features, positions of features, weather conditions, lighting conditions, or the like. Because the synthetic training data may be generated with predefined or known parameters, the features depicted or represented by the training data may be more accurate or precisely defined.
[0319] In some examples, the synthetic training data may be generated based on the captured training data, such as image frames of an environment and corresponding velocity and location data captured from a prior flight of an aerial vehicle 102, 200. In such examples, a mapping framework may generate a three-dimensional representation of the environment and / or a synthetic representation of an execution of a flight plan in the environment. The mapping framework may generate the three-dimensional representation by mapping the captured image frames based on the location data corresponding to the image frames. In some examples, the mapping framework may-81- 4910-9932-4237\3Docket Number P312830.W0.01 generate three-dimensional representations of objects by sampling the captured image frames based on the location data. For example, in generating a three-dimensional representation of a building, the mapping framework may generate the representation based on only image frames captured within a threshold distance from the building. In this manner, the mapping framework reduces the computer memory and processing capacity required to generate the representation as image frames captured beyond the threshold distance are not utilized to generate the representation, and the mapping framework increases the accuracy of the representation by generating the representation with only image frames captured near the building.
[0320] The mapping framework may generate the synthetic representation of an execution of a flight plan based on the captured image data and the velocity data.Sampling rates of captured or synthetic training data representative of an environment may be varied to model changes in velocity or direction in a real world environment. For example, images from a set can be skipped to simulate an increased velocity, or sampling rates can be slowed to simulate slower flight speeds. Where the prior flight was executed with a slower velocity than the velocity at which an autonomous flight of an aerial vehicle 102, 200 (e.g., an unmanned aerial vehicle) operates, the mapping framework may sample the captured image frames at a lower rate to simulate the faster velocity of an autonomous flight. Where the prior flight executed a flight plan at lower velocity (e.g. one-third the velocity) at which an autonomous flight operates, the mapping framework may sample at a rate or from the data by a scaled difference (e.g. only every third image) of the data captured by the prior flight when generating synthetic training data in order to simulate an autonomous flight at the increased velocity (e.g. at three times the velocity). Such training data helps to provide more realistic functionality by the autonomous system as the data will be more aligned with data captured in real time (e.g.. less detailed and missing information due to the velocity7of real time delivery and flight).
[0321] The training data may be annotated or tagged. The annotations or tags may define associations between or in addition to the features or environments represented by the training data. In some examples, the two-dimensional image data may be associated with LiDAR data or known depth information for the environments. The LiDAR data may be included in the training data or information in addition to the training data. For example, training data of various environments including images may be captured with-82- 4910-9932-4237\3Docket Number P312830.W0.01 LiDAR sensor information or other depth information. The known depth information can be associated with the images to train the models to recognize patterns between image data and distances to objects. The depth information may be included in or added to the training data for the landability model 1100, depth estimation model 1160 and / or the semantic understanding model 1180. In other examples, a depth or LIDAR reading of a location may be separately associated with images or other captured information of the location. In examples including synthetic training data, the synthetic training data may be generated including depth information or otherwise at known or predefined distances.
[0322] In various examples, the training data may additionally or alternatively include annotations including categorizations for various objects or features within the two-dimensional images. Such annotations may be used to train the semantic understanding model 1180 to perform semantic segmentation on two-dimensional images, or as additional contextual information for the depth estimation model 1160. The categorizations of objects and distances can be used to train the landability model 1100 to predict patterns between distances or sizes of various objects to assist in modeling or interpreting an environment. The categorizations may define all or a portion of individual features, or portions of the represented environment.
[0323] One or more networks are trained using the training data at block 3040. The one or more networks may be machine learned models or other artificial intelligence models such as neural networks, classifiers, or other types of models. The models may be trained using supervised or unsupervised learning techniques. For example, the depth estimation model 1160 may be trained utilizing two-dimensional image data and / or depth data corresponding to true distances of features, such as from LIDAR, radar, or the like or from annotations as training data. The semantic understanding model 1180 may be trained using supervised learning techniques and utilizing the two-dimensional image data, annotated tags or characteristics of the two-dimensional image data, and / or depth estimations as training data. The landability model 1100 can be trained on either or both sets of training data (e.g. object classification or depth related data). During training, the models may be trained to recognize or associate image data with types of or specific objects, discrete or ranges of distances, or various other features of the environment.
[0324] In various examples, the landability model 1100, depth estimation model 1160 and the semantic understanding model 1180 may be trained using compute resources-83- 4910-9932-4237\3Docket Number P312830.W0.011020 separate from the aerial vehicle 102, 200. After training, the machine learned models may be deployed as trained models. The machine learned models or modules may be stored on or in operative communication with the aerial vehicles 102, 200. In such examples, the trained depth estimation model 1160 and semantic understanding model 1180 may be provided to compute resources 1020 (e.g., through a wired or wireless communication interface to the aerial vehicle 102, 200).
[0325] At block 3060, the one or more trained networks are utilized to navigate an additional aerial vehicle 102, 200. The additional aerial vehicle 102, 200 can be the primary 102 or secondary aerial vehicle 200. The trained networks may be one or more of the compute resources 1020. For example, the trained networks may be the landability' model 1100. depth estimation 1160 or semantic understanding model 1180, the 3D world modeling 1280, the trajectory planner 1300 or the like. The aerial vehicle 102, 200 may utilize the trained networks or ML models to navigate a flight path to a delivery location or to a payload location at a delivery' location. For example, the secondary aerial vehicle 200 may navigate a descent and to deliver a payload in various environments.
[0326] To navigate the aerial vehicle 102, 200 the trained models or compute resources 1020 may interpret or identify the features of the delivery environment utilizing the trained networks. Information may be captured by one or more sensors 1040 for analy sis by the trained networks to determine the flight paths, follow flight paths, or to update flight paths. The trained models may analyze the captured information from the sensors 1040 to identify features or characteristics of an environment. For example, image data may be analyzed by the landability model 1100, depth estimation model 1160 or the semantic understanding model 1180 to identify or predict objects, such as obstacles or features, or their positions. In some examples, the distances or objects are not individually identified and rather patterns of objects at various distances relevant to navigation are identified. Obstacles or features, and their respective distances, may be associated with a payload location or flight paths, such as by the models 1100, 1160, 1180 or additional computing resources 1020, such as 3D world modeling 1280. The compute resources, such as the mission planner 1320 or the trajectory planner 1300. may generate or validate flight paths to navigate through the environment, or generate various commands to control features of the aerial vehicles 102, 200 to follow flight paths. For-84- 4910-9932-4237\3Docket Number P312830.W0.01 example, commands may be communicated to the GNC 1200 systems to autonomously navigate the aerial vehicles 102, 200 along a flight path.
[0327] FIG. 13 illustrates an example method 4000 of navigating a secondary aerial vehicle 200 from a primary aerial vehicle 102 to a delivery location. At block 4020, the aerial vehicle system 100 receives a delivery location. The deliver}' location may be specified by the system 100, or received from a shipper or delivery recipient. The delivery location may be communicated to the aerial vehicles 102, 200 as a coordinate or geographically define location, or the delivery location may be analyzed to determine or identify relevant information to communicate to the aerial vehicles 102, 200. The delivery location may be, in various examples, an address, GPS coordinates, selection within an image of a larger area, or the like. The delivery location may include or define a region or space to place a payload 101, such as a payload location. In some examples, a delivery recipient may provide additional information regarding the deliver}' location, such as a delivery time window or range. The deliver}7location may be received before or during navigation of the aerial vehicles 102, 200. For example, the aerial vehicles 102, 200 may be deployed after receipt of a delivery location, or the aerial vehicles may be in flight and receive or generate a flight path to the deliver}' location.
[0328] The secondary vehicle 200 may obtain a GPS or satellite determined location relative to the primary vehicle 102 at block 4040. The GPS location may be determined by one or more of the sensors 1040. The GPS location may be determined before, after, or during deployment of the secondary aerial vehicle 200 from the primary aerial vehicle 102. For example, the secondary vehicle 200 may be deployed by the primary vehicle 102 to navigate to the deliver}7location, such as at a payload location. The secondary vehicle 200 may, before, during, or after deployment, receive or determines a first or initial GPS location. The first GPS location of the secondary vehicle 200 may be compared to a known location or GPS location of the primary vehicle 102. The comparison may determine tolerances or ranges in the GPS locations, confirm or validate either location, or the like.
[0329] In some examples, the secondary vehicle 200 may utilize the GPS location of the primary aerial vehicle in combination with one or more sensors to determine a GPS location of the secondary aerial vehicle 200. In some examples, the secondary vehicle 200 may utilize the known location of the primary vehicle 102 during descent to-85- 4910-9932-4237\3Docket Number P312830.W0.01 determine its own location in an environment. For example, the GPS location of the primary vehicle 102 may be used by the secondary vehicle 200 as a reference point in determining GPS location of the secondary vehicle 200. In some examples, the GPS or satellite location of the secondary vehicle 200 may be independently determined or modeled within the environment.
[0330] The various compute resources 1020 may predict or determine a location of the secondary vehicle 200 in the environment based in part on the satellite or GPS location. For example, information captured by sensors 1040 may be compared to known map data by one or more of the compute resources 1020 to correct a satellite determine location.
[0331] At block 4060, the secondary vehicle 200 navigates to the delivery location using the GPS or satellite location and input from at least two sensors 1040 of the secondary vehicle 200. During descent, the secondary vehicle 200 may utilize data from sensors 1040 to determine its position relative to the delivery environment or a location of the primary vehicle 102. The secondary vehicle 200 may utilize sensor fusion to obtain a more accurate GPS location relative to the primary vehicle 102. For example, satellite coordinates may be analyzed by compute resources 1020 to correct for tolerances in the satellite determined coordinates of the aerial vehicles 102, 200.
[0332] In some examples, the sensors 1040 of the secondary vehicle 200 may capture information about motion of the vehicles 102, 200 provided by positional or inertial measurement sensors, image data from cameras, or the like. In other examples, different combinations of sensors 1040 may be used, such as sensors 1040 of the primary aerial vehicle 102 or sensors positioned at the delivery' location. In some examples, the two or more sensors 1040 may be dispersed between the primary 102 and the secondary’ aerial vehicle 200. For example, the sensors 1040 of the primary aerial vehicle 102 may capture information related to the positions of the secondary aerial vehicle 200. The secondary' vehicle 200 may utilize different combinations of the sensors 1040 for this purpose, such that the secondary' vehicle 200 may continue to navigate accurately in case of failure of one or more of the sensors 1040.
[0333] The compute resources 1020 may analyze the GPS and / or other sensor 1040 information corresponding to the environment to navigate the secondary' aerial vehicle 200. In some examples, multiple features or potential payload or landing locations may-86- 4910-9932-4237\3Docket Number P312830.W0.01 be associated with a single GPS determined location. By capturing information with the sensors 1040 in addition to a satellite location, the multiple features may be identified or more precisely associated with a unique position or location. For example, image data may be analyzed by the semantic understanding model 1180 or the depth estimation model 1160 to identify features of the environments. A landability model 1100 can utilize image data to model the environment, or associate characteristics of objects and distance information to identify airspace available for navigation. In some examples, the compute resources 1020, such as the 3D world modeling 1280 or localization 1260, may identify an actual position of the secondary vehicle 200 based on the outputs of the compute resources 1020, the GPS, or sensor information. For example, the identified features may be compared to known features at a delivery location. The compute modules 1020, such as the trajectory planner 1300, may generate commands to the GNC 1200 components to continue to follow a flight path, or to follow a new flight path to the delivery location if the actual position is different from the delivery' location.
[0334] Such navigation using sensor fusion may be used by the secondary vehicle 200 to precisely deliver to delivery locations when compared to use of GPS data by itself. For example, visual input or image data can correct GPS data or improve the precision of collected GPS data, giving the secondary vehicle 200 the ability to deliver more accurately to the delivery location (e.g., with error margins up to 10 cm rather than meters). Such accuracy is helpful where deliveries occur in crowded and / or small areas. For example, a payload is less likely to be delivered on the wrong side of a fence or to a wrong address.
[0335] Turning to FIG. 14, a flow chart illustrating an example method 6000 for navigating an aerial vehicle system 100 for a delivery’ is shown. The method 6000 may include step 6010 including receiving a delivery request identifying a delivery location. The delivery request may identify a payload to pick up from or deliver to the delivery location. The delivery’ request may be generated by one or more components of the system 100, such as the fleet management system 114, flight controller 116, or the mission planner 1320. The delivery request may be a user input to the system 100 by a user or delivery recipient, such as by a remote device 112. In some examples, the delivery request may be generated by a shipper or seller of goods. For example, the delivery request may be a product order or delivery instructions for a payload 101. The-87- 4910-9932-4237\3Docket Number P312830.W0.01 delivery request may identify a payload location, such as a region to place the payload 101 at the delivery location. In some examples, the delivery request may identify, or the delivery location may correspond to, map or satellite coordinates. The deliver}’ request, the delivery location, and / or payload location may be defined by natural language text or a structured input (e.g., formatted string or coordinates). The delivery request may identify alternative payload locations, or locations to avoid delivery' of a pay load 101. For example, a residential address may include public or occupied areas likely to be in use where a payload 101 or aerial vehicle 102, 200 may not be desired or designated to receive other shipments or mail.
[0336] In some examples, step 6010 may additionally or alternatively include navigating aerial vehicles from some initial location, such as an end location of another mission or delivery, to a shipper or aerial vehicle storage location. For example, the aerial vehicles 102, 200 may initially navigate to the shipper to receive the payload 101, or to return from delivery of a separate pay load 101.
[0337] The method 6000 may include step 6020 including navigating an aerial vehicle 102. 200 from a first location towards the delivery location. For example, the aerial vehicle 102, 200 may be the primary aerial vehicle 102 navigating over a distance, or the secondary aerial vehicle 200 navigating from the primary aerial vehicle 102 to the delivery location or payload location. The first location or delivery location may be a warehouse, restaurant, retail store, service center, residential location or a similar location where the aerial vehicle system 100 may be utilized. The navigation of the aerial vehicle 102, 200 may include traversal of a flight path between the first location and the delivery location. The flight path may be predetermined or generated autonomously by the aerial vehicle system 100. The aerial vehicle 102. 200 may navigate the flight path using one or more sensors 1040, such as a satellite receiver, cameras, or related components. The sensors 1040 may provide information to the aerial vehicles 102, 200 or a remote device, such as the flight controller 116, to generate or identify' the flight path. The compute resources 1020 may analyze the information captured by the sensors 1040 to interpret features of the flight path, update the flight path, or verify navigation along the flight path.
[0338] In some examples, step 6020 may additionally include performing a light housing maneuver during takeoff or during navigation of the aerial vehicle 102 from the-88- 4910-9932-4237\3Docket Number P312830.W0.01 first location. As disclosed herein, a light housing maneuver or other data collection maneuver is performed by the aerial vehicle 102, 200 such that the aerial vehicle 102, 200 rotates around an axis to change the position and / or otherwise changes orientations of various sensors 1040 to survey an area. For example, during takeoff the aerial vehicle 102, 200 may rotate 108 around the y-axis while ascending such that a forward-facing camera 174 positioned on or within the aerial vehicle 102, 200 may capture a multi directional views, e.g., views captured along the horizontal plane from a plurality of angles. The rotation may be 360 degrees or some subset thereof. Also, the rotation may be continuous or may be done at different times (e.g., at different z locations). However, by performing the maneuver at the same z location, the data captured may be consistent for that height, making the analysis faster. By performing a light housing maneuver, the aerial vehicle 102, 200 may enable the sensors 1040, including fixed sensors without individual movement capabilities, to collect data from a multitude of angles and directions. This may increase the sensory range and capability7of the aerial vehicle 102, 200 during takeoff which may, for example, increase the capability of the aerial vehicle 102. 200 to detect and avoid obstacles and intruder aircraft during takeoff.
[0339] In some examples, the aerial vehicle 102, 200 may perform a light housing maneuver during flight, e.g., the aerial vehicle 102, 200 may perform a light housing maneuver during traversal of the flight path between the first location and the delivery location to increase the sensory range and capability during flight. In a similar example, the aerial vehicle 102, 200 may perform a light housing maneuver while landing, e.g., the aerial vehicle 102, 200 may perform a light housing maneuver while descending to land or dock at a docking location.
[0340] The method 6000 may include step 6030 of determining an actual position of the aerial vehicle 102. 200 relative to the delivery location using refinement information. The aerial vehicle 102, 200 may initially determine a location by a satellite determined position. However, satellite determined locations may have tolerances that limit their effectiveness. For example, a delivery location may be a unit at an apartment including multiple units having a similar or same GPS coordinate. The refinement information may include information captured or determined by the sensors 1040, information generated by the compute resources 1020, such as the depth estimation model 1160 or semantic understanding model 1180, or predefined or existing information. For example, the-89- 4910-9932-4237\3Docket Number P312830.W0.01 refinement information may include captured images or image data captured by one or more cameras.
[0341] Information captured by the sensors 1040, such as images may include one or more pixels or features identifying or indicative of objects or characteristics of the environment. The features or objects of the environment may be identified or classified by the semantic understanding model 1180 and / or the depth estimation model 1160. In one example, the landability model 1100 can analyze capture images to predict or validate characteristics associated with the location of a delivery location. In some examples, the refinement information may include information from physical or inertial sensors. For example, movements or orientations of the aerial vehicle 102, 200 over a known distance along a route may assist in approximating or determining the actual position of the aerial vehicle 102, 200, or refining the outputs of the models.
[0342] In some examples, the refinement information may include or correspond to preexisting or known information regarding the delivery7location or flight path (e.g., map data, stored information about a delivery location, or the like). The preexisting information may include data gathered during separate or previous deployments to the delivery location or along a route to the delivery location.
[0343] In some examples, the refinement information may include one or more predetermined flight paths. The one or more predetermined flight paths may be associated with the first location or the delivery location. The predetermined flight paths may include instructions or information corresponding to maneuvers for the primary7aerial vehicle 102 to the delivery location, or for the secondary7aerial vehicle 200 from the primary7aerial vehicle 102 to the delivery location or pay load location. In some examples, the first location or the delivery7location may include one or more identifying characteristics or generate signals identified or received by the aerial vehicles 102. 200. The identifying characteristics or signals may be commands to execute predetermined flight plans or convey the predetermined flight plans to the aerial vehicle 102, 200.
[0344] To determine the position of the aerial vehicle 102, 200, the refinement information may be compared to estimate or received locational information, such as satellite determined coordinates, of the aerial vehicle 102, 200. For example, satellite, cellular, or other systems used to identify a location may have a tolerance or accuracy range, or existing information may include estimations or approximations of distances. In-90- 4910-9932-4237\3Docket Number P312830.W0.01 some examples, determining the actual position may be used to confirm or verify the position or location of the aerial vehicle 102, 200 determined by satellite or other positional sensors 1040. The aerial vehicle 102. 200 may be the primary vehicle 102 or the secondary vehicle 200. In some examples, the delivery location may be a region, rather than a discrete location. In such an example, the determining the actual position may include determining the position of the aerial vehicle 102, 200 within or relative to the delivery location.
[0345] In some examples, after generating, analyzing, or identifying refinement information, the refinement information may be compared to known information, such as related the estimated location of the aerial vehicles or delivery location, such as by the compute resources 1020. For example, features captured by the sensors 1040 or identified by the compute resources 1020 may be compared to known or predicted features of the flight path or delivery location. For example, a GPS coordinate may be associated with one or more features identified in map data, a delivery request, or the like. The compute resources 1020 may determine a similarity between the known features and captured refinement information. In some examples, various types of features may be w eighted as more reliable or predictive of an actual location, such as identified streets or buildings.
[0346] If the actual position corresponds to the delivery location the method 6000 may proceed to step 6040 and identify a pay load location at the deliver}’ location. If the actual position is different from the delivery location, the method 6000 may proceed to step 6036 and correct a position of the aerial vehicle 102, 200 from the actual position to the delivery location. In some examples, the aerial vehicle 102, 200 or system 100 may identify the actual position is likely a different location and determine a flight path to the delivery location. For example, the sensors 1040 may capture information about, or the compute resources 1020 may identify the aerial vehicle 102, 200 is at a location know n to the system 100 or an unknown location. If the aerial vehicle 102, 200 has not reached the delivery location, the aerial vehicle 102, 200 may continue to navigate to the delivery location by using the refinement information or by following the flight path. In some examples, the actual location may be verified as along the correct flight path but not at the delivery location. If the aerial vehicle 102, 200 is in the proximity of, but not at the delivery location, the refinement information may detect the delivery location or generate-91- 4910-9932-4237\3Docket Number P312830.W0.01 a flight path to the delivery7location. For example, the compute resources 1020 may generate commands for the propulsion systems 140, 220 to navigate or correct the flight path to the delivery location. After correcting the position of the aerial vehicle 102, 200. the method 6000 may proceed to step 6040.
[0347] At step 6040, the payload location at the delivery7location may be identified by the one or more sensors 1040 of the aerial vehicles 102, 200. For example, similar to step 6030, refinement information may be used to identify features of the delivery location. The refinement information may include captured images or image data captured by one or more sensors 1040, such as the cameras. The images may include information corresponding to one or more identifying obj ects or characteristics of the environment, which may be analyzed or determined by the semantic understanding model 1180 and / or the depth estimation model 1160. In some examples, a pattern of distances and objects associated with or likely to be suitable for a payload delivery^ can be identified by the landability model 1100. The prediction can optionally include or omit individual identification of objects or discrete distances of pixels of the image. In some examples, the refinement information may include information relating to a user input and / or delivery request. For example, the delivery request may provide coordinates or natural language text to assist in identifying the delivery location. In such an example, the compute resources 1020 may compare the identified features to the predicted or described features in the delivery request.
[0348] As discussed herein, the pay load location may be identified by determining whether various locations satisfy landability criteria for the secondary aerial vehicle 200 to land or deliver a payload 101 at a region of the delivery7location. For example, the refinement information may be used by the compute resources 1020 to identify objects near the location (e.g.. fences, power lines, people, and the like), flatness of the location, distance of the location to an initially selected location, surface type of the location (e.g., grass, dirt, concrete, and the like), or other criteria. The identified objects may be at or near the ground of the payload location (e.g., reachable by a person) or in the airspace above the payload location. In some examples, weather conditions may be factored into identifying a payload location. For example, if wind is present, a payload location having a tolerance in the direction of the wind may be preferable over an area with an object or-92- 4910-9932-4237\3Docket Number P312830.W0.01 obstacle downwind of the area. In some examples, the payload location may be identified before, during, or after deployment of the secondary aerial vehicle 200.
[0349] In some examples, the payload location may be identified based on a desirability ranking generated by the semantic understanding model 1180. The desirability ranking may rank possible payload locations captured by the one or more sensors 1040 at the delivery location. The ranking may be based on a semantic understanding of the features of the pay load location, customer and / or recipient feedback or other learning techniques regarding payload locations, and data of previous deliveries at payload locations. The semantic understanding model 1180 may iteratively update the desirability ranking over time based on further collected data and feedback.
[0350] In some examples, the landability model 1100 can identify the payload location without individually identifying or comparing factors relevant to determining a desirability ranking. For example, the landability model 1100 can be trained on data including such rankings and identity' patterns between regions satisfying a greater number of the factors. In such an example, the individual objects or distances can or may not be identified. Rather, captured data that could be used to identify objects or distances can be analyzed together to identity' locations predicted to be preferable (e g. highly ranked) as payload locations without requiring the individual identification of the objects or distances considered in the factors.
[0351] The method may optionally proceed to step 6050 including deploying a secondary aerial vehicle 200 from the aerial vehicle 102 to navigate to the payload location. In some examples, the secondary aerial vehicle 200 is not deployed and the primary' aerial vehicle 102 may navigate to the payload location. In other examples, the secondary vehicle 200 may be deployed prior to determining or identifying a payload location. For example, the payload location may be determined during navigation of the secondary vehicle 200.
[0352] To navigate to the payload location the aerial vehicle 102, 200 may follow a generated or secondary vehicle flight path to the payload location. In some examples, the secondary vehicle flight path is a delivery pathway. For example, a flight path from the primary aerial vehicle 102 to the payload location, or from a first or initial flight path to the payload location. The delivery pathway may include a vertical component, such as from a higher or lower elevation downward or upward, respectively. The delivery-93- 4910-9932-4237\3Docket Number P312830.W0.01 pathway may include a horizontal component corresponding navigable region of space horizontally at a given elevation or altitude. The secondary flight path may be generated by the compute resources 1020 at the aerial vehicles 102. 200 and / or a separate device. The secondary flight path may be a route to the payload location that avoids conflict with objects. In some examples, the secondary flight path may be generated or updated before or during deployment of the secondary aerial vehicle 200.
[0353] After arriving at the payload location, the secondary aerial vehicle 200 may land on or be positioned above a surface of the payload location. The secondary aerial vehicle 200 may deploy or release the payload 101. After releasing the payload 101, the primary or secondary7aerial vehicle 200 may navigate away or upward from the payload location.
[0354] In some examples, during the deployment of the secondary vehicle 200 to the payload location, the primary aerial vehicle 102 may perform a holding pattern to maintain a position relative to the payload location while the secondary7vehicle navigates to the pay load location, deploys the payload 101, and navigates from the payload location. The primary aerial vehicle may perform a light housing maneuver during the holding pattern to increase the detection range and capability of a sensor 1040 during the holding pattern. For example, while the secondary vehicle 200 is navigating to and from the payload location to deploy the payload 101, the primary7aerial vehicle 102 may hold a fixed position above the pay load location and rotate 108 around the y-axis to provide a sensor 1040 with a complete spherical view of the surroundings of the primary aerial vehicle 102.
[0355] Fig. 15 is a flow chart illustrating an example method 7000 for determining a payload location for delivery of a pay load by an aerial vehicle system 100. The example method 7000 may include step 7010 including navigating a primary aerial vehicle 102 to a delivery location. The navigation may be initiated by or responsive to a delivery request, or input to the system 100. In some examples, the navigation may be responsive to a request from within the system 100, such as by the fleet management system 114 or flight controller 116. As discussed herein, navigating may include determining the actual location of the aerial vehicle 102, 200 relative to the delivery location. The delivery location may be a warehouse, restaurant, retail store, service center, residential location or a similar location where the aerial vehicle system 100 may be utilized. For example,-94- 4910-9932-4237\3Docket Number P312830.W0.01 the delivery location may be a location where a payload 101 is delivered or received by the aerial vehicle 102, 200. The navigation of the aerial vehicle 102, 200 may include traversal of a flight path between the first location and the delivery location. The flight path may be predetermined or generated autonomously by the aerial vehicle system 100. The aerial vehicles 102, 200 may navigate to the delivery' location at least partially autonomously. For example, the aerial vehicle 102, 200 may navigate the flight path using one or more sensors 1040. The one or more sensors 1040 may gather refinement information including information about the delivery environment.
[0356] The example method 7000 may include step 7020 including identifying one or more objects at the delivery location. The objects may be identified by refinement information analyzed by one or more of the compute resources 1020. In some examples, objects may not be individually identified, and the compute resources (e.g. landability model 1100) may identify patterns in the refinement information associated with or summarizing the presence of objects.
[0357] The refinement information may include information gathered by the sensors 1040, preexisting information, or information generated by one or more components of the system 100. The preexisting information may be stored information about the delivery location, such as from previous flights. The preexisting information may be information regarding patterns or similar objects to identify objects at the location by comparison. Information gathered by the sensors 1040 may include types or classes of features, shapes, or distances of objects, as may be determined by analysis of image data. Information gathered by the sensors 1040 may be used by the compute resources 1020 to generate additional refinement information. The sensors 1040 of the aerial vehicles 102, 200 may capture information about the delivery location to detect objects before, during, or after arrival at the delivery location.
[0358] The information may be captured by either or both of the aerial vehicles 102, 200. For example, with reference to the primary aerial vehicle 102, after arriving at the delivery location, the primary aerial vehicle 102 may transition from forward flight to hover flight. In hover flight the primary aerial vehicle 102 may conduct a light housing maneuver to rotate 108 about an axis 106, as shown in FIG. 1. During rotation about the axis 106, the primary aerial vehicle 102 may capture refinement information by the one or more sensors 1040. For example, the primary' aerial vehicle 102 may use the visual or-95- 4910-9932-4237\3Docket Number P312830.W0.01 audio sensors 1040 to detect or determine the presence of aerial or ground based objects. By rotating the aerial vehicle 102. the sensors 1040 may continuously, or periodically, check the surroundings of the delivery location for incoming or moving objects. The changes of position or perspectives of the sensors 1040 may capture information for the determination of a source of a sound or light emitter, such as by determining changes in intensity of light or sound in various directions during rotation.
[0359] The primary aerial vehicle 102 may capture information regarding the secondary aerial vehicle 200. For example, the primary aerial vehicle sensors 1040 may determine or capture information such as a position of the secondary aerial vehicle 200 or whether a payload 101 has been released. While objects may be identified prior to deploying a secondary aerial vehicle 200. additional objects may be identified, or identifications of objects may be updated, during navigation by the secondary aerial vehicle 200. For example, the secondary aerial vehicle 200 may capture images by the sensors 1040, or other refinement information, during deployment from the primary aerial vehicle 102.
[0360] The example method 7000 may include step 7030 including identifying a payload location at the delivery location. The payload location may be a portion of the delivery location meeting one or more landing or landability criteria. The landability criteria may be determined in part by one or more of depth estimation model 1160, semantic understanding model 1180, three-dimensional world modeling 1280. trajectory’ planning 1300, mission planning 1320, or other components of the compute resources 1020 or the system 100. In some examples, the landability model 1100 can identify patterns or identify the presence of features in captured data to identify the payload location with or without individually analyzing the landability criteria. For example, the model can be trained to identify a combination of features predicted to satisfy landability criteria.
[0361] The landability’ criteria may be determined based on, or pay load locations may characteristically be identified based on, obstacles or objects near the location (e.g., fences, power lines, people, and the like), flatness of the location, distance of the location to an initially selected location, surface type of the location (e g., grass, dirt, concrete, and the like), recipient or user preference, or other criteria. For example, the payload location may be determined based on one or more of the presence of objects, or the-96- 4910-9932-4237\3Docket Number P312830.W0.01 locations of the objects. The various factors may be weighted. For example, a user preferred payload location may be weighted higher over a payload location having a better surface type. In some examples, a location may be flat and a preferred surface, but require navigation of the aerial vehicles 102, 200 in close proximity to objects and therefore avoided. In some examples, multiple payload locations may be identified and compared to determine a payload location satisfying more criteria, or more heavily weighted criteria. In some examples, after a payload location is identified the aerial vehicle system 100 may cease to identify additional locations.
[0362] The example method 7000 may include step 7040 including determining a flight path or delivery pathway from the primary aerial vehicle 102 to the payload location. The delivery pathway may be a route or path for the secondary aerial vehicle 200 to a ground or landing location. The delivery pathway may be defined to prevent contact with objects. In some examples, the delivery pathway may be defined to keep a determined distance away from an object or maintain a determined volume of open space for navigation. The delivery pathway may be defined with reference to the location of the primary aerial vehicle 102 relative to the payload location. The delivery pathway may define or include information identifying the maneuvers of the secondary aerial vehicle 200 to reach the payload location. The delivery pathway may be determined autonomously, such as by the compute resources 1020. In some examples, the delivery' pathway may be determined by the aerial vehicles 102. 200 or a device in communication with the aerial vehicles 102, 200 such as the flight controller 116 or fleet management system 114.
[0363] The delivery pathway may be determined before or during deployment of the secondary aerial vehicle 200. For example, in some examples, during traversal of the delivery pathway, the aerial vehicles 102. 200 may continue to capture information by the sensors 1040 to update the delivery pathway or the payload location. For example, the sensors 1040 may identify' information regarding objects or weather conditions, or changes thereof, at the payload location.
[0364] In some examples, a delivery pathway may be identified before or during identification of a payload location. For example, the number or duration of maneuvers required to traverse a deliver}' pathway may be relevant to the landability criteria. In-97- 4910-9932-4237\3Docket Number P312830.W0.01 some examples, fewer or shorter durations of maneuvers may be preferred to limit noise or power consumption.
[0365] The example method 7000 may include step 7050 including navigating the secondary aerial vehicle 200 from the primary aerial vehicle 102 along the delivery pathway. The navigation of the secondary’ aerial vehicle 200 may be controlled by the primary 102 and / or secondary aerial vehicle 200. For example, the primary' aerial vehicle 102 may control the tether 168 dispensed by the retraction assembly 160 or change a position, thereby changing the position of the secondary aerial vehicle 200. The secondary aerial vehicle 200 may control its position relative to the payload location or the primary' aerial vehicle 102 by the thrusters of the propulsion system 220.
[0366] During navigation of the delivery pathway, the secondary’ aerial vehicle 200 may capture additional information by the sensors 1040. The additional information may be used to validate the movements of the aerial vehicle 200, determine additional information about the delivery' environment, or the like. In some examples, the delivery’ pathway may be updated based on additional information captured by the sensors 1040. For example, additional objects or features may be identified that were previously obscured, distant, or otherwise not detected. In such an example, additional maneuvers may be generated or planned. After arriving at the payload location, the secondary' vehicle 200 may deploy a payload 101 for delivery.
[0367] Turning to Fig. 16, a flow chart illustrating an example method 8000 for navigating an aerial vehicle system 100 is shown. The method 8000 may include step 8010 including determining a first position or location of the aerial vehicle 102, 200. The first position may be determined in part by a satellite location, such as a GPS determined position. For example, the sensors 1040 may receive locational information from a satellite or cellular network identifying the position. The first position may be determined by' information captured by the one or more sensors 1040 of the aerial vehicles 102, 200. The sensors 1040 may be visual sensors 1040 and identify one or more characteristics of the position of the aerial vehicle 102, 200, such as the presence of a structure or objects. The one or more characteristics captured by the sensors 1040 may be associated or correspond to a known locations or positions relative to a location. For example, the various compute resources 1020 of the aerial vehicle system 100 may analyze the captured Information to determine the location of the aerial vehicle 102, 200.-98- 4910-9932-4237\3Docket Number P312830.W0.01
[0368] The first position may correspond to a position of the aerial vehicles 102, 200 which may be in flight or stationery. For example, the first position may be where the primary aerial vehicle 102 is hovering, navigating over a distance, stationary, or docked. In some examples, the first position may correspond to the secondary aerial vehicle 200. For example, the first position may be a location traversed during deployment of the secondary aerial vehicle 200.
[0369] The method 8000 may include step 8020 including determining the first position is in proximity to a known location. In some examples, the first position is at the know n location. The know n location may be a warehouse, shipper, restaurant, delivery location, residential location, or the like. In some examples, the known location may include one or more landing structures or payload locations. For example, one or more landing pads, docks, building entrances, or the like may be at the know n location. The known location may be identified prior to determining the first position, or after. For example, the known location may be an intended destination or an encountered location.
[0370] To determine the first position is in proximity to the known location, the gathered information, or information identifying the first location, may be analyzed. The known location may be determined in part by a satellite location, such as a GPS determined position. For example, the sensors 1040 may receive information from a satellite or cellular network identifying a coordinate or other locational information. The known location may be identified by information captured by the one or more sensors 1040 of the aerial vehicles 102, 200. In some examples, the captured information may be compared to preexisting information. For example, information gathered by the sensors 1040 may be compared to information at a database or ser er 118 including information about various locations. The determination may be by the aerial vehicles 102, 200, or by a remote device, such as the flight controller 116. The machine learned models or compute resources 1020 may determine a identify objects or characteristics at the position and that are associated with the known location.
[0371] In some examples, the known location may include an identifier associated with or identifying the known location. In some examples, the known location may be identified by features or structures at the location. The structures or fiducials at the location may passively identify the known location or features at the known location. The identifier may be passive by conveying information without establishing-99- 4910-9932-4237\3Docket Number P312830.W0.01 communication with the aerial vehicles 102, 200. For example, the identifier maybe markings associated with known information. In some examples, the markings may be defined by lights or movable features to convey multiple sets of information. In such an example, the sensors 1040 may capture information, such as images, of the environment to identify the features or location. For example, the compute modules 1020, such as the semantic understanding model 1180 may identify the passive identifier or structure.
[0372] In some examples, the identifier may be active (e.g., emitting or producing the signal). For example, an active identifier may generate an audio and / or electromagnetic signal (e.g., radio waves, Wi-Fi, Bluetooth) or visual signal. The fiducials may include information about the known location. The fiducials or identifiers may identify' features or objects present at the known location, such as payload locations or landing structures. For example, the fiducials may be associated with flight paths, environmental models, coordinates, or the like.
[0373] The known location may be associated with a predetermined flight path to a second position at the known location. For example, the information conveyed by the identifier may be associated with a predetermined flight path. In some examples, the identifier may be fiducial may identify a structure, such as a landing structure, and the know n location may be a delivery' location and the second location may be a payload or ground location. The known location may be a docking location where the second location is a flight corridor away from the docking location, or the second location is a dock or ground location for the aerial vehicle 102, 200. The predetermined flight path may be or include routes or identified navigable spaces about the known location. The predetermined flight path may include all or a portion of maneuvers or orientations for the aerial vehicle 102, 200 to navigate to the second location.
[0374] The predetermined flight path may be communicated to the aerial vehicles 102, 200 after identification of the known location. For example, the predetermined flight path may be identified by a component of the aerial vehicle system 100 before, during, or after arrival at the known location by the aerial vehicle 102, 200. The predetermined flight paths may be stored on the aerial vehicles 102, 200 and identified by the fiducials or known location. In some examples, the fiducials may communicate all or a portion of the predetermined flight path to the aerial vehicle 102, 200. For example, patterns or appearances of the fiducials, or signals generated by the fiducials, may identify' the flight-100- 4910-9932-4237\3Docket Number P312830.W0.01 paths or transfer information associated with the flight paths to the aerial vehicles 102, 200.
[0375] By generating or communicating a predetermined flight path, the aerial vehicle 102, 200 may navigate without or with reduced use of the sensors 1040 or processing demands. For example, the predetermined flight path may include a route around any features or objects of the known location. In such an example, the infrastructure or objects of the known location may be effectively ignored or known.
[0376] In some examples, multiple landing locations or landing structures may be at a single location. For example, multiple landing pads or docks may be at a given location.A GPS location may lack sufficient refinement to identify an intended landing pad or dock. By using a predetermined flight path, an aerial vehicle 102 may identify and navigate to the intended location.
[0377] By predetermining the flight path, increasingly complex maneuvers or closer travel to obstacles may be performed by the aerial vehicle 102, 200. Accordingly, navigation in congested areas may be performed. Further, the reduction of processing demands or sensor usage, may assist in quicker navigation of the aerial vehicle 102. 200.
[0378] In some examples, the method 8000 may include step 8030 including generating a flight path from the first position to the second position based on the predetermined flight path. The generated flight path may include a portion of a route after, before, or including the predetermined flight path. For example, the generated flight path may include one or more maneuvers to move at least one of the aerial vehicles 102, 200 from the first position to or about the known location. In some examples, the predetermined flight path may provide a starting point for a generated flight path, such as an intended route. The aerial vehicle system 100 may update or generate the remainder of the flight path. For example, weather conditions or changes to the maneuvers based on a payload weight may be accounted for by the system 100. In some examples, the sensors 1040 may verify or determine whether obstacles are present.
[0379] The method 8000 may include step 8040 including navigating the aerial vehicle 102, 200 of the aerial vehicle system 100 from the first position to the second position. For example, the primary aerial vehicle 102 may then navigate to the second position. In some examples, the secondary' aerial vehicle 200 may deploy from the primary' aerial-101- 4910-9932-4237\3Docket Number P312830.W0.01 vehicle 102 to navigate to the second position, where the primary aerial vehicle 102 may be the first position.
[0380] FIG. 17 is a schematic diagram of an example computer system 5000 for implementing various embodiments in the examples described herein. The computer system 5000 may be included in one or more components of the system 100. The computer system 5000 may be used to implement or operate the compute resources 1020. The computer system 5000 may be utilized to train or generate the ML models, such as the depth estimation model 1160 or the semantic understanding model 1180. For example, the compute resources 1020 may include one or more of the components of the computer system 5000 shown in FIG. 17. The computer system 5000 may be used to implement or execute one or more of the components or operations, such as those disclosed with reference to FIGS. 11-16. The computer system 5000 may include one or more processing elements 5020, an input / output interface 5040, a display 5060, one or more memory components 5080, a network interface 5100, and one or more external devices 5120. Each of the various components may be in communication with one another through one or more buses, communication networks, such as wired or wireless networks.
[0381] The processing element 5020 may be any type of electronic device capable of processing, receiving, and / or transmitting instructions. For example, the processing element 5020 may be a central processing unit, microprocessor, processor, or microcontroller. Additionally, it should be noted that some components of the computer 5000 may be controlled by a first processor and other components may be controlled by a second processor, where the first and second processors may or may not be in communication with each other.
[0382] The memory components 5080 are used by the computer 5000 to store instructions for the processing element 5020, as well as store data, such as the compute resources 1020 or the like. The memory7components 5080 may be, for example, magneto-optical storage, read-only memory, random access memory, erasable programmable memory, flash memory, or a combination of one or more types of memory components. The memory components 5080 may be included in the aerial vehicles 102, 200 or in operative communication with the aerial vehicles 102, 200.-102- 4910-9932-4237\3Docket Number P312830.W0.01
[0383] The display 5060 provides visual feedback to a user. Optionally, the display 5060 may act as an input element. The display 5060 may be a liquid crystal display, plasma display, organic light-emitting diode display, and / or other suitable display. In embodiments where the display 5060 is used as an input, the display may include one or more touch or input sensors, such as capacitive touch sensors, a resistive grid, or the like.
[0384] The I / O interface 5040 allows a user to enter data into the computer 5000, as well as provides an input / output for the computer 5000 to communicate with other devices or sendees. For example, an I / O interface 5040 to the compute resources 1020 may be used by a user to provide inputs, training data, sensor information, and / or other data to the compute resources 1020 for use in navigation of the aerial vehicle 102, 200. The I / O interface 5040 can include one or more input buttons, touch pads, and so on.
[0385] The network interface 5100 provides communication to and from the computer 5000 to other devices. For example, the network interface 5100 may be used by the compute resources 1020 to communicate with the aerial vehicles 102, 200, other devices of the aerial vehicle system 100, and / or with other computing systems. The network interface 5100 may include one or more communication protocols, such as, but not limited to Wi-Fi, Ethernet, Bluetooth, and the like. The network interface 5100 may include one or more hardwired components, such as a Universal Serial Bus (USB) cable, or the like. The configuration of the network interface 5100 depends on the types of communication desired and may be modified to communicate via Wi-Fi. Bluetooth, and so on.
[0386] The external devices 5120 are one or more devices that can be used to provide various inputs to the computing device 5000, e.g., mouse, microphone, keyboard, trackpad, or the like. The external devices 5120 may be local or remote and may vary as desired. In some examples, the external devices 5120 may include the one or more sensors 1040 of the aerial vehicles 102, 200.
[0387] The technology described herein may be implemented as logical operations and / or modules in one or more systems. The logical operations may be implemented as a sequence of processor implemented steps directed by software programs executing in one or more computer systems and as interconnected machine or circuit modules within one or more computer systems, or as a combination of both. Likewise, the descriptions of various component modules may be provided in terms of operations executed or effected-103- 4910-9932-4237\3Docket Number P312830.W0.01 by the modules. The resulting implementation is a matter of choice, dependent on the performance requirements of the underlying system implementing the described technology. Accordingly, the logical operations making up the embodiments of the technology described herein are referred to variously as operations, steps, objects, or modules. Furthermore, it should be understood that logical operations may be performed in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.
[0388] In some implementations, articles of manufacture are provided as computer program products that cause the instantiation of operations on a computer system to implement the procedural operations. One implementation of a computer program product provides a non-transitory computer program storage medium readable by a computer system and encoding a computer program. It should further be understood that the described technology may be employed in special purpose devices independent of a personal computer.
[0389] The above specification, examples and data provide a complete description of the structure and use of exemplary embodiments of the invention as defined in the claims. Although various embodiments of the claimed invention have been described above with a certain degree of particularity, or with reference to one or more individual embodiments, it is appreciated that numerous alterations to the disclosed embodiments without departing from the spirit or scope of the claimed invention may be possible. Other embodiments are therefore contemplated. It is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative only of particular embodiments and not limiting. Changes in detail or structure may be made without departing from the basic elements of the invention as defined in the following claims.
[0390] Other examples and implementations are within the scope and spirit of the disclosure and appended claims. For example, features implementing functions may also be physically located at various positions, including being distributed such that portions of functions are implemented at different physical locations. Also, as used herein, including in the claims, “or” as used in a list of items prefaced by "at least one of indicates a disjunctive list such that, for example, a list of “at least one of A, B, or C” means A or B or C or AB or AC or BC or ABC (i.e., A and Band C). Further, the term-104- 4910-9932-4237\3Docket Number P312830.W0.01‘‘exemplary ” does not mean that the described example is preferred or better than other examples.
[0391] The foregoing description, for purposes of explanation, uses specific nomenclature to provide a thorough understanding of the described embodiments.However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the described embodiments. Thus, the foregoing descriptions of the specific embodiments described herein are presented for purposes of illustration and description. They are not targeted to be exhaustive or to limit the embodiments to the precise forms disclosed. It will be apparent to one of ordinary skill in the art that many modifications and variations are possible in view of the above teachings.-105- 4910-9932-4237\3
Claims
Docket Number P312830.W0.01CLAIMSWhat is claimed is:
1. A method of navigating an aerial vehicle system for a delivery of a pay load comprising:navigating a primary' aerial vehicle of the aerial vehicle system to a delivery area; analyzing data including information from one or more sensors in communication with the aerial vehicle system to identify a payload location within the delivery area based on a landability assessment; anddeploying the payload to the payload location.
2. The method of claim 1, wherein the payload location is a region of the delivery’ area defined by one or more of objects or terrain sufficient to land an aerial vehicle of the aerial vehicle system or place the payload.
3. The method of claim 1, wherein deploying includes:navigating a secondary’ aerial vehicle of the aerial vehicle system dependently coupled with the primary aerial vehicle to the payload location,wherein the secondary’ aerial vehicle supports the pay load.
4. The method of claim 3, further comprising:analyzing refinement information corresponding to the delivery area, wherein the data includes the refinement information.
5. The method of claim 4, wherein:the refinement information includes data corresponding to a user input; and the user input identifies the delivery^ area.
6. The method of claim 5, wherein the user input identifies a preferred pay load location or regions to avoid at the delivery area.
7. The method of claim 4, wherein:-106- 4910-9932-4237\3Docket Number P312830.W0.01 the refinement information includes the data from the one or more sensors of the aerial vehicle system; andthe data includes information corresponding to one or more objects or characteristics of the delivery area.
8. The method of claim 4, further comprising:determining an actual position of the primary aerial vehicle relative to the delivery area using the refinement information;navigating the primary aerial vehicle from the actual position to the delivery area.
9. The method of claim 8, further comprising:identifying the actual position by capturing the refinement information by the one or more sensors; andcomparing the captured refinement information with preexisting information associated with the delivery area.
10. The method of claim 4, further comprising:determining a secondary' vehicle flight path from the primary' aerial vehicle to the payload location using the refinement information; andwherein the refinement information includes one or more predetermined flight paths associated with the delivery area.
11. The method of claim 3, further comprising:capturing images of the delivery area by the one or more sensors, wherein the data includes the images;analyzing the images by a machine learned model to determine one or more delivery' pathways to the payload location; andwherein the delivery pathways are based at least in part on a presence or position of one or more of objects, features, or characteristics of the delivery area.
12. The method of claim 11, wherein the delivery' pathway is a route from the primary aerial vehicle to the payload location without contacting one or more obstacles-107- 4910-9932-4237\3Docket Number P312830.W0.01 based on the presence or position of the one or more objects, features, or characteristics of the delivery area.
13. The method of claim 11, wherein the machine learned model to identifies a distance corresponding to a portion of the delivery area by analyzing the data.
14. The method of claim 11, wherein the machine learned model identifies the one or more objects, features, or characteristics of the deli very’ area by analyzing the data.
15. The method of claim 11, wherein:the machine learned model includes:a first machine learned model identifying the one or more objects, features, or characteristics of the delivery area, anda second machine learned model determining a distance corresponding to the one or more objects, features, or characteristics of the delivery area.
16. The method of claim 3, further comprising:delivering the payload at the payload location, wherein the secondary' aerial vehicle lands on or is positioned above the payload location and releases the payload.
17. The method of claim 1, wherein the one or more sensors are separately located from the primary aerial vehicle or the information from the one or more sensors is captured prior to navigation of the primary' aerial vehicle.
18. The method of claim 1, wherein the data is provided to the aerial vehicle system prior to navigation of the primary aerial vehicle.
19. The method of claim 1, wherein the data is communicatively received by the aerial vehicle system during navigation of the primary aerial vehicle.
20. A method of determining a payload location for delivery of a payload by an aerial vehicle system comprising:-108- 4910-9932-4237\3Docket Number P312830.W0.01 navigating a primary aerial vehicle of the aerial vehicle system to a delivery' location responsive to a delivery’ request;identifying one or more objects at the delivery location;identifying the payload location at the delivery location, wherein the payload location is a portion of the delivery location meeting one or more landability criteria; determining a delivery pathway from the primary aerial vehicle to the payload location; andnavigating a payload package from the primary aerial vehicle along the delivery pathway.
21. The method of claim 20, wherein:navigating comprises utilizing a secondary aerial vehicle, andthe payload package is releasably coupled to the secondary’ aerial vehicle.
22. The method of claim 21, further comprising:after navigating the primary aerial vehicle to the delivery location, rotating the primary aerial vehicle or the secondary aerial vehicle about an axis to search for one or more aerial objects at the delivery' location.
23. The method of claim 22, wherein:the axis is a vertical axis, andthe primary aerial vehicle rotates about the vertical axis while the secondary aerial vehicle is deployed from the primary aerial vehicle.
24. The method of claim 21. further comprising:monitoring a position of the secondary aerial vehicle relative to the primary aerial vehicle during navigation of the delivery’ pathway.
25. The method of claim 24, wherein:the primary aerial vehicle includes one or more sensors, andthe one or more sensors capture images of the secondary aerial vehicle to determine the position of the secondary vehicle relative to the primary aerial vehicle.-109- 4910-9932-4237\3Docket Number P312830.W0.0126. The method of claim 21. further comprising:capturing refinement information by one or more sensors of the aerial vehicle system.
27. The method of claim 26, wherein:spaces occupied by one or more objects are identified by a first machine learned model by analyzing the refinement information captured by the one or more sensors of the aerial vehicle system.
28. The method of claim 27, wherein a second machine learned model determines a distance corresponding to the objects by analyzing the refinement information captured by the one or more sensors.
29. The method of claim 28, wherein:the first machine learned model or the second machine learned model are trained by analyzing training data; andthe training data includes one or more annotations identifying portions of the training data representative of environmental objects similar to the objects or distances to the environmental objects.
30. The method of claim 29, wherein the training data comprises one or more of captured images or computer generated images.
31. The method of claim 28. wherein navigating the secondary aerial vehicle comprises:determining the distances to the objects and selecting a pathway through an at least partially vertical space between the primary aerial vehicle and the payload location.
32. The method of claim 21, further comprising:-110- 4910-9932-4237\3Docket Number P312830.W0.01 delivering the payload at the payload location, wherein the secondary' aerial vehicle lands on or is positioned above a feature at the payload location and releases the payload.
33. A method of navigating an aerial vehicle comprising:determining a first position of the aerial vehicle;determining the first position is in proximity to a known location by detecting a fiducial at the known location by one or more sensors of the aerial vehicle, wherein the fiducial is associated with a predetermined flight path to a second position;generating a flight path from the first position to the second position based on the predetermined flight path; andnavigating the aerial vehicle from the first position to the second position.
34. The method of claim 33, wherein:the first position is determined by the one or more sensors.
35. The method of claim 33, wherein:the one or more sensors include a camera to capture the fiducial, andthe fiducial is a first fiducial corresponding to a first portion of the predetermined flight path and a second fiducial corresponds to a second portion of the predetermined flight path.
36. The method of claim 33, wherein the fiducial is in communication with the aerial vehicle by one or more of visible indicators, electromagnetic emissions, or acoustic emissions.
37. The method of claim 33, wherein:during navigation of the generated flight path the aerial vehicle limits or reduces capturing of information by the one or more sensors.
38. The method of claim 33, wherein the first position is a docking location and the second position is a flight corridor.-Il l- 4910-9932-4237\3Docket Number P312830.W0.0139. The method of claim 33, wherein the first position is a flight corridor and the second position is a payload delivery location or a docking location.
40. A method of navigating an aerial vehicle of an aerial vehicle system comprising:determining a first position of the aerial vehicle;identifying a known location by one or more sensors of the aerial vehicle system, wherein the known location is associated with one or more objects with known positions at the known location;determining the first position is in proximity' to or at the known location; generating a flight path from the first position to a second position based at least in part on one or more predetermined flight paths associated with the known location to avoid the one or more objects; andnavigating the aerial vehicle from the first position to the second position.
41. The method of claim 40, wherein:during navigation of the generated flight path the aerial vehicle limits or reduces capturing of information by the one or more sensors.
42. The method of claim 40, wherein navigating from the first position to the second position comprises:deploying a secondary' aerial vehicle from a primary aerial vehicle at the first position, wherein the aerial vehicle is the secondary' aerial vehicle, andnavigating the secondary’ aerial vehicle from the primary aerial vehicle to the second position.
43. The method of claim 40, w herein the first position is a docking location and the second position is a flight corridor.
44. The method of claim 40, wherein the first position is a flight corridor and the second position is a payload delivery location or a docking location.-112- 4910-9932-4237\3Docket Number P312830.W0.0145. A method of delivering a payload:navigating a primary unmanned aerial vehicle to a delivery area; determining a delivery pathway from the primary unmanned aerial vehicle to a payload location within the delivery area; andnavigating a secondary unmanned aerial vehicle from the primary unmanned aerial vehicle downwards within the delivery pathway to the payload location.
46. The method of claim 45, wherein the secondary unmanned vehicle is a dependent vehicle to the primary unmanned aerial vehicle.
47. The method of claim 45, further comprising:capturing refinement information corresponding to the delivery area by sensors coupled to the primary unmanned aerial vehicle or the secondary unmanned aerial vehicle; andidentifying, by at least a first machine learned model while the primary unmanned aerial vehicle is at the delivery' area, one or more environmental features at the delivery area or distances corresponding to the one or more environmental features.
48. The method of claim 47, wherein:the first machine learned model identifies the one or more features at the delivery area from data generated from the refinement information corresponding to the one or more environmental features; anda second machine learned model identifies the distances associated with the data from the refinement information corresponding to the one or more environmental features.
49. The method of claim 45, wherein the delivery pathway is determined, at least in part, by:identifying a pattern between a presence of and distance to environmental obstacles at the delivery area to identify the payload location, anddetermining a route between the primary unmanned aerial vehicle and the payload location avoiding contact with the identified environmental obstacles.-113- 4910-9932-4237\3Docket Number P312830.W0.0150. The method of claim 45. wherein the payload location is identified based, at least in part, on a presence of environmental objects between the primary unmanned aerial vehicle and a portion of the delivery area.
51. The method of claim 45, wherein the payload location is determined by a presence of or distance to environmental obstacles at the delivery area having a primary purpose separate from an aerial vehicle system.
52. The method of claim 45, wherein the primary unmanned aerial vehicle navigates to the delivery area by information captured by one or more sensors during navigation.
53. The method of claim 45, wherein the primary unmanned aerial vehicle navigates to the delivery area utilizing information generated prior to navigation.
54. The method of claim 45, wherein the primary unmanned aerial vehicle navigates to the delivery area by one or more of information communicated to the primary unmanned aerial vehicle during navigation, information captured by one or more sensors during navigation, or information generated prior to navigation.
55. A method of training a machine learned model to analyze an environment comprising:generating training data including information representative of the environment, wherein at least a portion of the training data is computer generated;assigning one or more annotations to the training data identifying one or more portions of the training data representative of at least one characteristic of the environment;training the model to identify the at least one representative characteristic by analyzing the training data; anddeploying a trained model in operative communication with an aerial vehicle system to identify an environmental characteristic of a real world environment corresponding to the at least one characteristic.-114- 4910-9932-4237\3Docket Number P312830.W0.0156. The method of claim 55. wherein at least a second portion of the training data includes information captured by one or more sensors representative of the real world environment.
57. The method of claim 55, wherein:the at least one characteristic is a classification of a feature of the environment; andthe annotation is an identifier to assist in classifying the feature.
58. The method of claim 57. wherein the trained model identifies one or more environmental objects positioned in the real world environment based on captured information corresponding to the at least one characteristic.
59. The method of claim 55, wherein:the at least one characteristic is a depth or distance to a feature of the environment; andthe annotation is a true depth or distance corresponding to one or more portions of the training data including the feature.
60. The method of claim 59, wherein the trained model identifies depths or distances to one or more environmental objects in the real world environment based on captured information corresponding to the at least one characteristic.
61. The method of claim 55, wherein inputs for training the trained model include refinement information captured by one or more sensors of an unmanned aerial vehicle.
62. The method of claim 55, wherein:the training data includes information corresponding to an output from a second trained model; andthe annotation is an identification of at least a portion of the output relevant to determining the characteristic.-115- 4910-9932-4237\3Docket Number P312830.W0.0163. A method of navigating an aerial vehicle comprising:capturing information of an environment by one or more sensors: identifying a passive identifier corresponding to information about the environment;generating a flight path from a first position to a second position based on the information about the environment; andnavigating the aerial vehicle from the first position to the second position.
64. The method of claim 63. wherein:the passive identifier is associated with a docking structure, andthe information about the environment includes a location of the docking structure.
65. The method of claim 64. wherein:the docking structure is one of a plurality of docking structures, andthe passive identifier identifies the docking structure from the plurality of docking structures.
66. The method of claim 65, further comprising:analyzing the captured information from the one or more sensors by one or more machine learned models to identify the passive identifier.
67. The method of claim 66, wherein:the machine learned models are stored at a memory component onboard the aerial vehicle.
68. The method of claim 21, wherein navigating the primary aerial vehicle of the aerial vehicle system to the delivery location comprises rotating the primary rotating the primary aerial vehicle or the secondary aerial vehicle about an axis during takeoff to search for one or more aerial objects during ascension.-116- 4910-9932-4237\3Docket Number P312830.W0.0169. The method of claim 21. wherein navigating the primary aerial vehicle of the aerial vehicle system to the delivery location comprises rotating the primary rotating the primary aerial vehicle or the secondary aerial vehicle about an axis during docking to search for one or more aerial objects during descension.
70. The method of claim 21, wherein navigating the primary aerial vehicle of the aerial vehicle system to the delivery location comprises rotating the primary rotating the primary aerial vehicle or the secondary aerial vehicle about an axis during execution of a flight plan to search for one or more aerial objects along the flight plan.
71. The method of claim 55, wherein the information captured by the one or more sensors representative of the real world environment comprises image frames, location data, and velocity data captured during a prior flight of an aerial vehicle in the real world environment.
72. The method of claim 55, wherein generating training data including information representative of the environment comprisesgenerating synthetic training data representing an execution of a flight plan by an unmanned aerial vehicle based on the image frames, location data, and velocity' data simulating a capture during a prior flight, comprising:determining a difference in velocity between the simulated prior flight and an expected velocity of the unmanned aerial vehicle based on the captured velocity data;sampling the captured image frames at a sampling rate based on the difference in velocity; andgenerating a synthetic representation of the execution of the flight plan by the unmanned aerial vehicle based on the sampled captured images.
73. The method of claim 55, wherein generating training data including information representative of the environment comprises generating synthetic training data of a three-dimensional representation of the environment based on image frames, location data, and velocity data captured during a prior flight, comprising:-117- 4910-9932-4237\3Docket Number P312830.W0.01 identifying an object in the real world environment based on the image frames; identifying a location of the object based on the location data;identifying a first image frame captured within a threshold distance of the object based on the location data; andgenerating a three-dimensional representation of the object based on the first image frame.
74. The method of claim 20, wherein identifying the payload location at the delivery location comprises:identifying one or more potential locations at the delivery location that meet the one or more landability criteria;generating a ranking of the one or more potential locations based on a ranking preference; andidentifying a highest ranked potential location as the payload location.
75. The method of claim 74. wherein generating the ranking of the one or more potential locations based on the ranking preference comprises:receiving a user input comprising a preference for the payload location; updating the ranking preference based on the user input; andranking the one or more potential locations based on the updated ranking preference.
76. The method of claim 74, wherein generating the ranking of the one or more potential locations based on the ranking preference comprises:identifying a semantic feature of the one or more potential locations, wherein the semantic feature comprises at least one of a geometry of the one or more potential locations, a classification of the one or more potential locations, or a classification of an object identified at the one or more potential locations;ranking the semantic feature of the one or more potential locations based on the ranking preference; andranking the one or more potential locations based on the ranking of the semantic feature.-118- 4910-9932-4237\3Docket Number P312830.W0.0177. The method of claim 74. wherein navigating the payload package from the primary aerial vehicle along the delivery pathway further comprises:delivering the payload package at the identified payload location;receiving a user input comprising feedback regarding the delivery' of the payload package at the identified payload location; andupdating the ranking preference based on the user input.
78. The method of claim 1, comprising:capturing images of the delivery area; andanalyzing the captured images by a machine learned model to identity the payload location based at least in part on distance information and a type of object at the delivery area obtained from the captured images.
79. The method of claim 78. wherein the machine learned model identifies the payload location based on a pattern between the type of object and the distance information.
80. The method of claim 20. comprising:capturing images of the delivery location;analyzing the captured images by a machine learned model to identify the one or more objects and associate the objects with distance information; andidentifying the pay load location by the machine learned model based on a pattern between the objects and the distance information.
81. The method of claim 80, wherein the machine learned model is trained to identify the payload location based on an association between the pattern and the one or more landability criteria.
82. The method of claim 55, whereinthe at least one characteristic of the environment includes an object classification and distance information to the object, and-119- 4910-9932-4237\3Docket Number P312830.W0.01 the model is trained to identify a location in the real world environment based at least on patterns between the object classification and the distance information.
83. The method of claim 82. wherein after deployment of the trained model, the trained model analyzes captured images data of the real world environment to identify a pay load location based at least in part on similarities to the patterns between the object classification and the distance information to patterns in the training data between similar features.
84. A method for delivering a payload by an aerial vehicle system comprising:capturing an image of a delivery area by a sensor of an aerial vehicle; analyzing the image by a machine learned model trained to identify a payload location within the delivery area based on a landability assessment; anddeploying a dependent vehicle operatively associated with the aerial vehicle to deliver the payload to the payload location.
85. The method of claim 84, wherein the payload location is a region of the delivery area defining a stable area to land the dependent vehicle or release the payload.
86. The method of claim 84, wherein analysis by the machine learned model includes one or more of determining distance or classifications of objects captured within the image.
87. The method of claim 84, wherein the machine learned model is trained to predict the payload location by analyzing the images to identify a pattern between objects present at the delivery area and distances to the objects.
88. The method of claim 87, further comprising:analyzing the images by the machine learned model to determine one or more delivery pathways to the payload location, the delivery pathways defined as a volume of airspace navigable by at least the dependent vehicle between the aerial vehicle and the payload location.-120- 4910-9932-4237\3Docket Number P312830.W0.0189. A method of delivering a payload, comprising:capturing an image of a delivery area by a sensor:analyzing the image by a machine learned model trained to predict a payload location within the delivery area based on patterns between at least spaces occupied by objects and distances to the spaces occupied by objects within the delivery area;directly predicting the payload location from by the machine learned model; and delivering the payload to the payload location.
90. The method of claim 89, wherein the payload location is predicted without independently identifying the objects occupying the spaces or the distances to the objects.
91. The method of claim 89, wherein the image is captured before or during a delivery operation for delivery’ the payload.
92. The method of claim 89, wherein two images are captured from at least two different orientations, and the distance to spaces occupied by the objects are inferred by comparing differences between the different orientations.
93. The method of claim 89, wherein the machine learned model is trained by analyzing training data including information corresponding to classifications of objects or distances to objects to predict or identify the patterns based on patterns in the training data between similar features.
94. The method of claim 89, comprising:generating, by the machine learned model, a three dimensional model of the delivery area to predict the payload location.-121- 4910-9932-4237\3
