Using a reproducible real-time processing graph for autonomous vehicle operations

Abstract

In some embodiments, a computer-implemented method of generating perception commands is provided. A simulation computing system determines delay prediction information for a perception system. The delay prediction information is usable to predict an amount of delay for an autonomous vehicle computing system to generate perception commands using an in-flight perception graph. The simulation computing system obtains perception data, and provides the perception data to a simulation perception graph to generate a perception command. The perception command includes a timestamp and an indication of one or more types of perception data used by the simulation perception graph to generate the command. The simulation computing system determines a predicted delay for the perception command based on the delay prediction information. The simulation computing system updates the timestamp of the perception command based on the predicted delay. The simulation computing system provides the perception command and the predicted delay to a simulation queue.

Description

USING A REPRODUCIBLE REAL-TIME PROCESSING GRAPH FOR AUTONOMOUS VEHICLE OPERATIONSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S Application No. 18 / 902,226, filed on September 30, 2024, the contents of which are incorporated herein by reference.TECHNICAL FIELD

[0002] This disclosure relates generally to computer perception, and in particular but not exclusively, relates to improved perception architectures for use with autonomous vehicles.BACKGROUND

[0003] Development of control systems for autonomous vehicles is a challenging task. Control systems are often created and tested using development machines and simulations that have different processing power and / or capabilities than the autonomous vehicle systems that will operate the control systems during actual use. Further, while sensor data collected by autonomous vehicles can be logged during use, the processing of the sensor data may be non-deterministic due to lossy algorithms, non-deterministic dropping of data due to processing delays, or for other reasons. Successful development of autonomous control systems should allow for as much determinism and test fidelity as possible, which existing techniques currently lack. What is desired are new development techniques and architectures that provide as much determinism and fidelity to actual operating environments as possible.BRIEF SUMMARY

[0004] In some embodiments, a non-transitory computer-readable medium having computer-executable instructions stored thereon is provided. The instructions, in response to execution by one or more processors of a simulation computing system, cause the simulation computing system to perform actions for generating perception commands, the actions comprising: determining, by the simulation computing system, delay prediction information for a perception system, wherein the delay prediction information is usable to predict an amount of delay for an autonomous vehicle computing system to generate perception commands using an in-flight perception graph; obtaining, by the simulation computing system, perception data; providing, by14049-P103WOthe simulation computing system, the perception data to a simulation perception graph to generate a perception command, wherein the perception command includes a timestamp and an indication of one or more types of perception data used by the simulation perception graph to generate the command; determining, by the simulation computing system, a predicted delay for the perception command based on the delay prediction information; updating, by the simulation computing system, the timestamp of the perception command based on the predicted delay; and providing, by the simulation computing system, the perception command and the predicted delay to a simulation queue.

[0005] In some embodiments, a computer-implemented method of generating perception commands is provided. A simulation computing system determines delay prediction information for a perception system. The delay prediction information is usable to predict an amount of delay for an autonomous vehicle computing system to generate perception commands using an in-flight perception graph. The simulation computing system obtains perception data, and provides the perception data to a simulation perception graph to generate a perception command. The perception command includes a timestamp and an indication of one or more types of perception data used by the simulation perception graph to generate the command. The simulation computing system determines a predicted delay for the perception command based on the delay prediction information. The simulation computing system updates the timestamp of the perception command based on the predicted delay. The simulation computing system provides the perception command and the predicted delay to a simulation queue.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0006] Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. Not all instances of an element are necessarily labeled so as not to clutter the drawings where appropriate. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles being described. To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced.24049-P103WO

[0007] FIG. 1 is a schematic diagram that illustrates a system in which computer perception techniques are developed and used for autonomous vehicle navigation, according to various aspects of the present disclosure.

[0008] FIG. 2A and FIG. 2B illustrate an aerial vehicle or UAV, in accordance with various embodiments of the present disclosure.

[0009] FIG. 3 is a block diagram that illustrates further components of a nonlimiting example embodiment of an autonomous vehicle according to various aspects of the present disclosure.

[0010] FIG. 4 is a block diagram that illustrates aspects of a non-limiting example embodiment of a simulation computing system according to various aspects of the present disclosure.

[0011] FIG. 5 is a schematic illustration of interactions between a perception engine and a simulation engine, according to various aspects of the present disclosure.

[0012] FIG. 6 is a schematic illustration of a non-limiting example embodiment of a perception graph, according to various aspects of the present disclosure.

[0013] FIG. 7 is a schematic illustration of the processing of two input data streams according to various aspects of the present disclosure.

[0014] FIG. 8A - FIG. 8B are a flowchart that illustrates a non-limiting example embodiment of a method of generating perception commands, according to various aspects of the present disclosure.DETAILED DESCRIPTION

[0015] Perception software applications are often expressed as graphs (i.e., vertices / nodes representing compute tasks and edges representing data connections). This offers a very natural software development paradigm since relationships between various components are generally well-represented as a graph. The processing behavior of graph libraries typically operate as either "offline" or "real-time." Offline processing does not aim to make any timing guarantees, and instead aims to produce a specific set of outputs given a specific set of inputs regardless of input data arrival time or of processing time. This is know n as "determinism. " In contrast, real-time processing aims to meet timing guarantees. In the event of unexpectedly -high-rate input data or slow processing times, timing guarantees may not be able to be met. A common way of dealing with this is by dropping data, opting for incomplete output in favor of falling34049-P103WOirrecoverably behind. Since the output may vary from one execution to the next (even with the same inputs), this is considered "non-deterministic."

[0016] When supporting the operation of autonomous vehicles, a real-time graph is typically preferred, or else data outputs may be too old to be useful to a closed-loop autonomous control system). However, there are reasons why determinism is desirable wherever possible, such as for the sake of simulation. Determinism provides reliable tests, since the results can be relied upon to be the same each time, and complicated statistical approaches for detecting root causes of errors are unnecessary. It also provides for reproducibility' in log replay, where the inputs provided to the graph from a real-time flight are used during offline execution. Log replay is useful to, for example, validate that a code change fixes a bug exhibited in some production mission.

[0017] Supporting log replay is one way that determinism can be provided. Determinism only expects the same results given the same inputs. Indeed, two identical real (i. e. , as opposed to simulated) flights are not expected to have the exact same results due to physical noise on sensors, changes in environmental conditions, and so on. As such, sensor data may be logged and used as an input. Then, the rest of the system can behave deterministically, and actions of the flight controller can be reproduced using the logged data. If logged, any sources of non-determinism can also be treated as inputs to achieve deterministic replay.

[0018] Timing is one source of non-determinism to be addressed. On the vehicle, frames should be processed in a timely fashion, or else the processing results become increasingly less valuable for use in real-time autonomous control. Unfortunately, no two executions will have the exact same timings, and fluctuations of available CPU resources will provide different timing from run-to-run. Further, issues may compound in the event of a slowdown: not only will future inputs change since outputs are used in a closed-loop system, but work may back up unbounded if inputs arrive faster than they can be processed. For this reason, input frames (which occur at a regular rate) may be dropped if they are not processed in a timely manner. While this is a non-deterministic event, it may be logged to reproduce it faithfully during log replay.

[0019] Another source of timing non-determinism is that of asynchronous programming. In order to meet deadlines, especially with long-running tasks, concurrency may be used to process incoming data. Even if the code is written to be thread-safe, it may not be deterministic. There are also other sources of non-determinism to consider; for example, random number generators and differences in 44049-P103WOhardware (e.g., CPU vs. GPU implementations) may be non-deterministic. Some may not even have feasible solutions. For example, log size constraints may lead to the use of lossy compression, or logging at a reduced rate. These limit the ability to reproduce results, but determinism can be relaxed in certain scenarios: for example, parts of the image processing pipeline may be bypassed when image logs are unavailable (e.g., bypass the fiducial detector and just use logged fiducial detections), or rely on a representative distribution (e.g., lossy images or random numbers may not reproduce perfectly, but with enough replays, issues should recur at the same rate as in production, and the average results produced should be roughly the same).

[0020] Techniques are described herein to provide graph perception processing that is (1) deterministic in simulation, (2) real-time in production, and (3) reproducible in log replay. While some existing graph processing libraries support one or two of these goals, no previous graph processing library is known to support all three goals. In some embodiments of the present disclosure, a graph processing library that supports both real-time and offline execution is updated, such that limited, selected points of non-determinism (e.g., places where perception data can be dropped in the event of a slowdown) are inserted to enable real-time performance. Any non-deterministic events that occur at these locations in the graph are logged in a way that can be reproduced faithfully during log replay.

[0021] Some embodiments of the present disclosure include one or more of: (1) automatic logging of arbitrary data streams (edges in the graph), including streams that note when data is being dropped from another stream; (2) automatic construction of graph elements depending on w hether the graph is being used in simulation mode or not (e.g., allow dropped frames and write which frames were dropped in production; read which frames were dropped in simulation and don't allow other dropped frames, etc.); (3) well-defined pattern of resetting algorithm state prior to performing a live mission so that, in order to successfully perform a log replay, the state of the autonomous vehicle does not have to be recorded, and / or the log replay does not have to commence from system startup; (4) well-defined patterns for using the deterministic ("offline") graph in ways that minimize latency (e g., ensuring sufficiently high-rate data where necessary, using the "latest" rather than the "best" match).

[0022] In some embodiments, "variable-depth" log replay is provided. In such embodiments, the framework supports replaying logs from intermediate / intemal edge logs (as opposed to input edge logs). This helps overcome technical problems in having 54049-P103WOlimited storage, as it may be impractical or impossible to store / log all input data due to its size. As such, log replays from intermediate data instead of input edge data can help overcome limitations in available storage space.

[0023] In some embodiments, delay reproduction / modeling is provided, such that non-determinism and delays experienced during real-time processing can be modeled during offline simulation processing. By logging timestamps and graph path information with every graph output, the deterministic nature of the graph may be leveraged to associate outputs with inputs for creating high-fidelity simulations.

[0024] FIG. 1 is a schematic diagram that illustrates a system in which computer perception techniques are developed and used for autonomous vehicle navigation, according to various aspects of the present disclosure. One or more autonomous vehicles 106 are deployed to operate autonomously in a given geographic area. In some embodiments, the autonomous vehicles 106 may be part of a fleet of autonomous vehicles managed to collectively accomplish tasks, including but not limited to collecting sensor data (e.g., aerial imagery’, etc.), performing package delivery, or transporting passengers.

[0025] Each of the autonomous vehicles 106 may include a variety of perception sensors, and information collected by the perception sensors may be used by the autonomous vehicle 106 to draw conclusions about the environment surrounding the autonomous vehicle 106 and / or the relationship of the autonomous vehicle 106 to its environment. These conclusions may then be used by the autonomous control systems of the autonomous vehicle 106 to help accomplish tasks assigned to the autonomous vehicle 106.

[0026] During operation, each autonomous vehicle 106 may generate logs of data generated by its perception sensors, results of perception processing performed on such data, and / or other data generated during autonomous operation. These logs may then be transmitted to a simulation computing system 102 via a network 104, which may use one or more of wireless communication technologies, wired communication technologies, or any other suitable technique for transferring the log information from the autonomous vehicles 106 to the simulation computing system 102. The simulation computing system 102 may be used to create, debug, update, and / or otherwise develop the perception software operated by the autonomous vehicles 106. The simulation computing system 102 may replay the logs in order to debug issues that may have arisen during flight, may use the logs to increase the fidelity of simulations, and / or may use 64049-P103WOthe logs in other ways to improve the development and performance of the perception systems as described in further detail below.

[0027] The autonomous vehicles 106 may include any type of vehicle configured to operate autonomously, including but not limited to one or more of unmanned aerial vehicles (UAVs), autonomous road vehicles (e.g., passenger cars, delivery vans, trucks, etc.), or other types of vehicles. While the majority of the description herein relates to UAVs for the sake of convenience, this should not be seen as limiting, and one of ordinary skill in the art will recognize that the disclosed techniques may be used with any type of autonomous vehicle.

[0028] As a non-limiting example of an autonomous vehicle. FIG. 2A and FIG. 2B illustrate an unmanned aerial vehicle or UAV 200. in accordance with an embodiment of the present disclosure. The illustrated embodiment of UAV 200 is a vertical takeoff and landing (VTOL) unmanned aerial vehicle (UAV) that includes separate propulsion units 212 and propulsion units 208 for providing horizontal and vertical propulsion, respectively. UAV 200 is a fixed-wing aerial vehicle, which as the name implies, has a wing assembly 224 that can generate lift based on the wing shape and the vehicle’s forward airspeed when propelled horizontally by propulsion units 212. FIG. 2A is a perspective top view illustration of UAV 200 while FIG. 2B is a bottom side plan view illustration of UAV 200.

[0029] The illustrated embodiment of UAV 200 includes a fuselage 220. In one embodiment, fuselage 220 is modular and includes a battery module, an avionics module, and a mission payload module. These modules are detachable from each other and mechanically securable to each other to contiguously form at least a portion of the fuselage 220 or UAV main body.

[0030] The battery module includes a cavity for housing one or more batteries for powering UAV 200. The avionics module houses flight control circuitry of UAV 200, which may include a processor and memory, communication electronics and antennas (e.g.. cellular transceiver, Wi-Fi transceiver, etc.), and various sensors (e.g., global positioning sensor, an inertial measurement unit (IMU). a magnetic compass, etc.). The mission payload module houses equipment associated with amission of UAV 200. For example, the mission payload module may include a payload actuator for holding and releasing an externally attached payload. In another embodiment, the mission payload module may include a camera / sensor equipment holder for carrying camera / sensor equipment (e.g., camera, lenses, radar, LIDAR, pollution monitoring 74049-P103WOsensors, weather monitoring sensors, etc.). Other components that may be carried by some embodiments of the UAV 200 (or other types of autonomous vehicles) are illustrated in FIG. 3.

[0031] The illustrated embodiment of UAV 200 further includes horizontal propulsion units 212 positioned on wing assembly 224, which can each include a motor, shaft, motor mount, and propeller, for propelling UAV 200. The illustrated embodiment of UAV 200 includes two boom assemblies 206 that secure to wing assembly 224.

[0032] The illustrated embodiments of boom assemblies 206 each include a boom housing 218 in which a boom is disposed, vertical propulsion units 208, printed circuit boards 216, and stabilizers 202. Vertical propulsion units 208 can each include a motor, shaft, motor mounts, and propeller, for providing vertical propulsion. Vertical propulsion units 208 may be used during a hover mode where UAV 200 is descending (e.g., to a delivery7location) or ascending (e.g., following a delivery ). Stabilizers 202 (or fins) may be included with UAV 200 to stabilize the UAV’s yaw (left or right turns) during flight. In some embodiments, UAV 200 may be configured to function as a glider. To do so. UAV 200 may power off its propulsion units and glide for a period of time.

[0033] During flight, UAV 200 may control the direction and / or speed of its movement by controlling its pitch, roll, yaw, and / or altitude. For example, the stabilizers 202 may include one or more rudders 204 for controlling the UAV’s yaw, and wing assembly 224 may include elevators for controlling the UAV’s pitch and / or ailerons 210 for controlling the UAV’s roll. As another example, increasing or decreasing the speed of all the propellers simultaneously can result in UAV 200 increasing or decreasing its altitude, respectively. The UAV 200 may also include components for sensing the environment around the UAV 200, including but not limited to audio sensor 222 and audio sensor 214. Further examples of sensor devices are illustrated in FIG. 3 and described below.

[0034] Many variations on the illustrated fixed-wing aerial vehicle are possible. For instance, aerial vehicles with more wings (e.g., an “x-wing” configuration with four wings), are also possible. Although FIG. 2A and FIG. 2B illustrate one wing assembly 224, two boom assemblies 206, two horizontal propulsion units 212, and six vertical propulsion units 208 per boom assembly 206, it should be appreciated that other variants of UAV 200 may be implemented with more or fewer of these components.84049-P103WOLikewise, as mentioned above an autonomous vehicle may take a form other than an aerial vehicle, such as a ground-based vehicle or water-based vehicle.

[0035] It should be understood that references herein to an autonomous vehicle can apply equally to autonomous and semi-autonomous vehicles. In a fully autonomous implementation, all functionality7of the vehicle is automated; e.g., preprogrammed or controlled via real-time computer functionality that responds to input from various sensors and / or pre-determined information. In a semi-autonomous implementation, some functions of a vehicle may be controlled by a human operator, while other functions are carried out autonomously. Further, in some embodiments, an autonomous vehicle may be configured to allow a remote operator to take over functions that can otherwise be controlled autonomously by the autonomous vehicle. Yet further, a given type of function may be controlled remotely at one level of abstraction and performed autonomously at another level of abstraction. For example, a remote operator may control high level navigation decisions for an autonomous vehicle, such as specifying that the autonomous vehicle should travel from one location to another (e.g., from a warehouse in a suburban area to a delivery address in a nearby city), while the autonomous vehicle's navigation system autonomously controls more fine-grained navigation decisions, such as the specific route to take between the two locations, specific flight controls to achieve the route and avoid obstacles while navigating the route, and so on.

[0036] FIG. 3 is a block diagram that illustrates further components of a nonlimiting example embodiment of an autonomous vehicle according to various aspects of the present disclosure. In some embodiments, the autonomous vehicle 302 is an aircraft such as the UAV 200 illustrated in FIG. 2A and FIG. 2B, though in other embodiments, the components illustrated in the autonomous vehicle 302 may be in a different type of vehicle (e.g., an automobile, etc.).

[0037] As shown, the autonomous vehicle 302 includes a communication interface 304, one or more vehicle state sensor devices 306, a power supply 308, one or more processors 310. one or more propulsion devices 312, one or more perception sensors 322, and a computer-readable medium 314. Since the autonomous vehicle 302 includes processors 310, communication interfaces 304, a computer-readable medium 314, etc., the autonomous vehicle 302 may be referred to as a computing system.

[0038] In some embodiments, the communication interface 304 includes hardware and software to enable any suitable communication technology for 94049-P103WOcommunicating with the simulation computing system 102 (or other computing systems). In some embodiments, the communication interface 304 includes multiple communication interfaces, each for use in appropriate circumstances. For example, the communication interface 304 may include a long-range wireless interface such as a 4G or LTE interface, or any other ty pe of long-range wireless interface (e.g., 2G, 3G, 5G, or WiMAX), to be used to communicate with the simulation computing system 102 while traversing a route. The communication interface 304 may also include a mediumrange wireless interface such as a Wi-Fi interface to be used when the autonomous vehicle 302 is at an area near a start location or an endpoint where Wi-Fi coverage is available. The communication interface 304 may also include a short-range wireless interface such as a Bluetooth interface to be used when the autonomous vehicle 302 is in a maintenance location or is otherwise stationary and waiting to be assigned a route. The communication interface 304 may also include a wired interface, such as an Ethernet interface or a USB interface, which may also be used when the autonomous vehicle 302 is in a maintenance location or is otherwise stationary and waiting to be assigned a route.

[0039] In some embodiments, the vehicle state sensor devices 306 are configured to detect states of various components of the autonomous vehicle 302, and to transmit signals representing those states to other components of the autonomous vehicle 302. Some non-limiting examples of vehicle state sensor devices 306 include a battery state sensor and a propulsion device health sensor.

[0040] In some embodiments, the power supply 308 may be any suitable device or system for storing and / or generating power. Some non-limiting examples of a power supply 308 include one or more batteries, one or more solar panels, a fuel tank, and combinations thereof. In some embodiments, the propulsion devices 312 may include any suitable devices for causing the autonomous vehicle 302 to travel along the path. For an aircraft, the propulsion devices 312 may include devices such as, but not limited to, one or more motors, one or more propellers, and one or more flight control surfaces.

[0041] In some embodiments, the processors 310 may include any type of computer processor capable of receiving signals from other components of the autonomous vehicle 302 and executing instructions stored on the computer-readable medium 314. In some embodiments, the processors 310 may include multiple processors, and / or one or more processors having multiple processing cores. In some 104049-P103WOembodiments, the processors 310 may include one or more general purpose processors and / or one or more processors adapted for a special purpose, including but not limited to a vision processing unit (VPU), a graphics processing unit (GPU), or a tensor processing unit (TPU).

[0042] In some embodiments, the computer-readable medium 314 may include one or more devices capable of storing information for access by the processors 310. In some embodiments, the computer-readable medium 314 may include one or more of a hard drive, a flash drive, an EEPROM, and combinations thereof.

[0043] In some embodiments, the one or more perception sensors 322 may include any ty pe of sensor for detecting characteristics of the environment of the autonomous vehicle 302, and / or the relationship of the autonomous vehicle 302 to its environment. In some embodiments, the perception sensors 322 may include one or more types of cameras for capturing imagery from the point of view of the autonomous vehicle 302. For example, the perception sensors 322 may include one or more of a downward-facing camera or an angled-view camera. In some embodiments, the one or more perception sensors 322 may include one or more cameras configured to capture different types of imagery, including but not limited to a visible light camera, an infrared camera, a light-field camera, an intensity' camera, a Tango camera (i.e., a camera that generates information including motion tracking information to track position and orientation), a laser camera, a stereoscopic camera, and / or a time-of-flight camera. The perception sensors 322 may also include a GNSS sensor, one or more accelerometers (and / or other devices that are part of an inertial navigation system), LIDAR devices, telemetry' sensors, and / or other sensor devices for sensing an environment of the autonomous vehicle 302 and / or the relationship of the autonomous vehicle 302 to its environment.

[0044] As shown, the computer-readable medium 314 has stored thereon a log data store 316, a data logging engine 318, a perception engine 324, and a route traversal engine 320. In some embodiments, the data logging engine 318 is configured to log various types of information generated by the autonomous vehicle 302, such as data captured by the perception sensors 322, data captured by the vehicle state sensor devices 306, and / or information generated by the perception engine 324, and to store the log data in the log data store 316. In some embodiments, the perception engine 324 is configured to receive perception data from the perception sensors 322 and to provide perception commands based on processing of the perception data to the route traversal 114049-P103WOengine 320 to aid in autonomous control of the autonomous vehicle 302. In some embodiments, the route traversal engine 320 is configured to cause the propulsion devices 312 to propel the autonomous vehicle 302 through a route based on perception commands received from the perception engine 324. The route traversal engine 320 may use the perception commands to assist in positioning and navigation as is typical for an autonomous vehicle.

[0045] Many different techniques may be used by the perception engine 324 to process the perception data to create perception commands. In a nai ve approach, the perception engine 324 may use a while loop to process each image frame and / or other type of perception data generated by the perception sensors 322. While such an approach may be comprehensive in addressing all of the collected perception data, there are nevertheless technical drawbacks to doing so. For example, due to varying processing loads on the processors 310, the perception engine 324 may not be able to keep up with the rate at which the perception data is generated by the perception sensors 322, and so the perception commands may begin to lag behind the actual state of the autonomous vehicle 302 and fail to achieve the real-time processing desired from the system. As another example of a technical drawback of this approach, it is very static and difficult to reconfigure, such that developing and debugging the perception engine 324 becomes very difficult.

[0046] In some embodiments, technical benefits are achieved by using a perception graph to represent the processing performed by the perception engine 324. In a perception graph, nodes represent processing performed on perception data, and edges between the nodes represent transmission of data (either input perception data from perception sensors 322. or intermediate data generated by nodes of the graph). Any suitable framework for implementing the perception graph may be used, including but not limited to the open source MediaPipe framework from Google LLC.

[0047] As discussed above, using a perception graph provides numerous technical benefits. For example, a perception graph architecture has a flexible foundation, making it easy to add, remove, or reconfigure perception data sources, processing algorithms, and sinks. As another example, frameworks such as MediaPipe allow for strong synchronization semantics, and support for state and state transitions. Further, a perception graph may be deployed in both operational environments (e.g., on the autonomous vehicle 302) and in simulation environments (as discussed in further detail below) with minimal changes, if any. Additional benefits of the perception graph 124049-P103WOused by embodiments of the present disclosure include the ability to inject delay into simulations and the ability to use the same perception graph for replayed log data and for simulation data, as will be discussed in further detail below.

[0048] As used herein, "engine" refers to logic embodied in hardware or software instructions, which can be written in one or more programming languages, including but not limited to C, C++, C#, COBOL, JAVA™, PHP, Perl. HTML, CSS, JavaScript, VBScnpt. ASPX, Go. and Python. An engine may be compiled into executable programs or written in interpreted programming languages. Software engines may be callable from other engines or from themselves. Generally, the engines described herein refer to logical modules that can be merged with other engines, or can be divided into sub-engines. The engines can be implemented by logic stored in any type of computer-readable medium or computer storage device and be stored on and executed by one or more general purpose computers, thus creating a special purpose computer configured to provide the engine or the functionality thereof. The engines can be implemented by logic programmed into an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or another hardware device.

[0049] As used herein, "data store" refers to any suitable device configured to store data for access by a computing device. One example of a data store is a highly reliable, high-speed relational database management system (DBMS) executing on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key-value store. However, any other suitable storage technique and / or device capable of quickly and reliably providing the stored data in response to queries may be used, and the computing device may be accessible locally instead of over a network, or may be provided as a cloud-based service. A data store may also include data stored in an organized manner on a computer-readable storage medium, such as a hard disk drive, a flash memory, RAM, ROM, or any other type of computer-readable storage medium. One of ordinary skill in the art will recognize that separate data stores described herein may be combined into a single data store, and / or a single data store described herein may be separated into multiple data stores, without departing from the scope of the present disclosure.

[0050] FIG. 4 is a block diagram that illustrates aspects of a non-limiting example embodiment of a simulation computing system according to various aspects of the present disclosure. The illustrated simulation computing system 102 may be implemented by any computing device or collection of computing devices, including 134049-P103WObut not limited to a desktop computing device, a laptop computing device, a mobile computing device, a server computing device, a computing device of a cloud computing system, and / or combinations thereof. The simulation computing system 102 is configured to receive, analyze, and process log data from autonomous vehicles 106 to support improvements to perception processing for autonomous operation.

[0051] As shown, the simulation computing system 102 includes one or more processors 402, one or more communication interfaces 404. a log data store 408, a delay data store 412, and a computer-readable medium 406.

[0052] In some embodiments, the processors 402 may include any suitable type of general-purpose computer processor. In some embodiments, the processors 402 may include one or more special-purpose computer processors or Al accelerators optimized for specific computing tasks, including but not limited to graphical processing units (GPUs), vision processing units (VPUs), and tensor processing units (TPUs).

[0053] In some embodiments, the communication interfaces 404 include one or more hardware and or software interfaces suitable for providing communication links between components. The communication interfaces 404 may support one or more wired communication technologies (including but not limited to Ethernet, FireWire, and USB), one or more wireless communication technologies (including but not limited to Wi-Fi. WiMAX, Bluetooth. 2G, 3G, 4G, 5G, and LTE), and / or combinations thereof.

[0054] As shown, the computer-readable medium 406 has stored thereon logic that, in response to execution by the one or more processors 402, cause the simulation computing system 102 to provide a log collection engine 410, a delay determination engine 414, a perception engine 418, and a simulation engine 416.

[0055] In some embodiments, the log collection engine 410 is configured to retrieve log information stored by one or more autonomous vehicles 106, and to store the retrieved information in the log data store 408. In some embodiments, the delay determination engine 414 is configured to analyze the information from the log data store 408 to determine predictions of processing delays (or other delays) that will likely be experienced when processing specific types of data to generate specific perception commands, and to store the predicted delay information in the delay data store 412. In some embodiments, the perception engine 418 is configured to use a perception graph to generate perception commands based on perception data, either from a replayed log or from simulated sources. In some embodiments, the simulation engine 416 is 144049-P103WOconfigured to simulate operation of an autonomous vehicle 106, and may also be configured to generate simulated input for the perception engine 418.

[0056] Further description of the configuration of each of these components is provided below.

[0057] FIG. 5 is a schematic illustration of interactions between a perception engine and a simulation engine, according to various aspects of the present disclosure. As shown, the perception engine 418 provides perception data from either a perception simulation 502, a log data store 408, or both, to the perception graph 506 for processing. The information from the log data store 408 provided to the perception graph 506 may include one or more of recorded perception data, indications of non-deterministic events such as dropped frames, intermediate results from the perception graph 506, telemetry information, or other logged data recorded by an autonomous vehicle 302 during a mission. Information provided from the perception simulation 502 to the perception graph 506 may include simulated data generated by the simulation engine 416, including but not limited to aerial imagery from a simulated location, simulated motion / attitude information, or simulated telemetry information for a simulated autonomous vehicle. The perception graph 506 generates a perception command 508, which is provided by the perception engine 418 to a simulation queue 510 of the simulation engine 416.

[0058] In order to compensate for the fact that the perception engine 418 executing on the simulation computing system 102 is unlikely to operate at the same speed as the perception engine 324 operating on the autonomous vehicle 302, and in order to therefore maintain timestamp synchronization between the perception engine 418 and the simulation engine 416, a vehicle simulation 504 may block execution until one or more perception commands 508 with appropriate timestamps are available in the simulation queue 510, at which point the vehicle simulation 504 advances the simulation clock and uses the perception commands 508 in the simulation queue 510 as input. In some embodiments, a "time of availability” or ‘"total delay from input timestamp” may be provided for each sink in the perception graph 506. Further, the perception graph 506 may be designed such that results advance through internals of the perception graph 506 deterministically, even if some components of the perception graph 506 operate significantly slower or faster on the simulation computing system 102 than on the autonomous vehicle 302.154049-P103WO

[0059] FIG. 6 is a schematic illustration of a non-limiting example embodiment of a perception graph, according to various aspects of the present disclosure. In FIG. 6, nodes / vertices that represent calculations are illustrated as squares. Input, intermediate, and output data is illustrated with a wavy line at the bottom side. Edges, or communication between nodes / vertices, are illustrated as arrows.

[0060] The sample perception graph 600 shows three simulated perception data input streams: a stream that provides simulated intensity images 604, a stream that provides simulated telemetry data 612, and a stream that provides simulated depth images 614. Each stream provides packets of data (e.g., images, etc.), and each packet is marked with a timestamp. The input streams are provided to one or more synchronizers 606, 608, which may be configured to provide input packets with matching timestamps to the rest of the sample perception graph 600 for processing. Since the input data streams may be operating at different refresh rates and / or with different amounts of latency, the use of synchronizers 606, 608 allows for a greater amount of determinism in the execution of the sample perception graph 600. Further details regarding the operation of the synchronizers 606, 608 are provided in the description of FIG. 7.

[0061] The sample perception graph 600 performs several non-limiting examples of processing of perception data. For example, the simulated telemetry data 612 and simulated depth image 614 is processed by a stereo AGL 616 (above ground level) node and a stereo obstacle 618 node. The stereo AGL 61 node is configured to use telemetry data and depth image data to estimate an altitude at which the autonomous vehicle 302 is present. The stereo obstacle 618 is configured to use telemetry data and depth image data to detect potential obstacles.

[0062] While the stereo AGL 616 and stereo obstacle 618 processing is relatively simple in that it is a single step computation, more complicated computations may be performed. For example, the sample perception graph 600 includes a semantic segmenter 610 node, which uses a simulated intensity image 604 and simulated telemetry data 612 to generate a generated semantic image 628 (e.g., an image that includes semantic labels on objects detected in the image, such as houses, cars, landing locations, power lines, etc.). The generated semantic image 628 is then provided to a semantic abort 622 node and a semantic nudge 624 node. The semantic abort 622 node may generate perception commands to cause the mission to be aborted in response to identifying particular objects. The semantic nudge 624 node may align detected objects 164049-P103WOin the perception data with known map locations in order to adjust a geolocation detected by less reliable sensors of the autonomous vehicle 302.

[0063] One technical advantage of the perception graph is that it allows for modification at several locations in order to support debugging, development, and other purposes. In some embodiments, the same sample perception graph 600 may be used to process either simulated perception data or logged perception data. For example, the synchronizer 606 may accept an input data stream that includes either simulated intensity images 604 or logged intensity images 602. In some embodiments, the sample perception graph 600 may also be pruned to use logged intermediate data instead of directly using the logged or simulated perception data. For example, a data stream that provides logged semantic images 620 may be used as input to the semantic abort 622 node and the semantic nudge 624 node instead of the generated semantic images 628 from the semantic segmenter 610. This can help provide technical benefits in situations where adequate storage space for a log of all of the input data is not available, or where it is desirable to avoid the computation time used for upstream operations within the perception graph. Though the sample perception graph 600 illustrates logged versus simulated data in only a couple of locations, one of ordinary skill in the art will recognize that other perception graphs may be more complicated, and / or may allo7for the substitution of simulated data for logged data at many different points within the perception graph.

[0064] Another technical advantage of the perception graph is that it allows for logging at multiple locations. For example, the input to and output from any of the nodes / vertices in the perception graph may be logged, along with a timestamp indicating a time at which the input or output took place. By performing such detailed logging, deterministic and faithful execution of log replays may be performed even when the simulation computing system 102 has different computing power than the autonomous vehicle 106.

[0065] Detailed logging can also help reproduce non-deterministic execution. For example, various nodes / vertices of the perception graph may be associated with input queues of configurable size. If the input queue for a given node / vertex reaches a (configurable) maximum queue size, the perception engine 418 may drop further input data until the input queue reaches a state where additional input may be accepted. Such frame-dropping is a non-deterministic event because pressure on the queue will depend on compute timing and performance, and so data logging should take this into account 174049-P103WOso that the handling of the dropped frames can be reproduced accurately during log replay.

[0066] Dropping data at source nodes / vertices provides a variety of benefits. For example, the non-deterministic event that is logged can simply be rolled into the non-deterministic arrival of frames from the source node or input stream. In other words, instead of separately logging which messages were dropped from which queue (then using those logs in replay), the input frames that are dropped may simply not be logged as input. As another example, by dropping frames at a source early in the traversal of the perception graph, computing time is not wasted. Dropped messages in the middle of a graph ty pically mean that the work leading up to that message was a waste of CPU time.

[0067] In some embodiments, parallel processing may be used for some or all of the components of the perception graph. For example, a separate thread may be used for each node / vertex in the perception graph. If the processors 402 of the simulation computing system 102 and / or the processors 310 of the autonomous vehicle 302 include a number of cores that matches the number of nodes / vertices in the perception graph, this can be particularly efficient. However, if fewer cores are available than there are nodes / vertices in the perception graph, the multiplexing abilities of a thread scheduler may be used, such that various threads may sleep to w ait for the results of long -running nodes / vertices in order to improve the throughput of the perception graph overall.

[0068] One aspect of reproducibility and determinism in perception graph processing is that, as multiple input streams are processed, the perception graph should handle the input data in a deterministic manner, even if the input data streams are slightly out of sync with each other. FIG. 7 is a schematic illustration of the processing of two input data streams according to various aspects of the present disclosure. In some embodiments, the functionality illustrated in FIG. 7 may be performed by a synchronizer 606, 608 within the perception graph. In other embodiments, the functionality illustrated in FIG. 7 may be performed prior to the data being provided to the perception graph.

[0069] Various rules may be applied to determine when a perception graph should begin processing a frame of input data from one or more input streams: (1) Packets from different input streams with matching timestamps will be processed together with a single Process invocation; (2) monotonicity of streams is preserved; (3) timestamps of available input frames have matching timestamps. These rules have 184049-P103WOseveral implications: A call to invoke processing on one or more input streams may be delayed until all streams have made sufficient progress so that the rules are met. To do so, packets from any input data stream may be empty, indicating that there will not be a packet with that timestamp from that input stream to avoid one inactive stream preventing processing from being called.

[0070] FIG. 7 illustrates times at which processing by the perception graph may be run. and paired inputs, for packets arriving offset in real time. As shown, packets from a first input stream (stream A) arrive with timestamps 100, 110, 120, and 130, while packets from a second input stream (stream B) arrive with timestamps 105 and 120. slightly offset from the real time at which the packets are received from the first input stream. Each row of the table indicates when the perception graph would process the data from the two input streams: for timestamps at which there is no matching data in the other input stream (or for which an unillustrated empty data packet is provided by the other input data stream), a data packet from a single input data stream is processed, whereas for timestamps at which there is matching data in the other input stream (i.e., timestamp 120), the perception graph processes both data packets together. At timestamp 130, the perception graph may be waiting to determine whether a packet wi th a matching timestamp will be received from stream B prior to processing the packet from stream A.

[0071] FIG. 8A - FIG. 8B are a flowchart that illustrates a non-limiting example embodiment of a method of generating perception commands, according to various aspects of the present disclosure. The method 800 illustrates at least one technical benefit of the architecture used in the present disclosure, in that the same method 800 may be used for either deterministic log replay or simulation without significant alterations to the method 800 or the architecture. When being used for log replay, the information recorded in the logs can help the method 800 to reproduce non-deterministic dropped frames. Meanwhile, for simulation, the method 800 can accurately reproduce typical processing delays, which is non-trivial since the simulation computing system 102 and the autonomous vehicle 106 typically offer widely different processing power for different tasks.

[0072] From a start block, the method 800 proceeds to a set of method steps 840 wherein log data is collected from one or more autonomous vehicles 106. Though illustrated with a single set of steps, in some embodiments, the set of method steps 840194049-P103WOmay be performed for multiple autonomous vehicles 106, consecutively and / or at least partially concurrently.

[0073] In the set of method steps 840, at block 802, an autonomous vehicle 106 uses one or more perception sensors 322 to collect perception data. In some embodiments, the perception sensors 322 generate perception data streams that include a time series of data points (e.g., image frames, sensor readings, etc.). The data points that make up a stream are referred to interchangeably herein as "‘frames,’7which may refer to image frames or data points of other types. Each data point may be labeled with a timestamp based on a shared clock provided by the autonomous vehicle 302, such that data points having matching timestamps were collected at the same time.

[0074] At block 804, a data logging engine 318 of the autonomous vehicle 106 stores the perception data in a log data store 316 of the autonomous vehicle 106. In some embodiments, the data logging engine 318 may store all data points in all of the perception data streams in the log data store 316. In other embodiments, the data points of the perception data streams may be stored when processed as an input or an output by some node / vertex of the perception graph. This will allow non-deterministic frame dropping to be reproduced, as the data logging engine 318 may store the processed frames without storing the dropped frames.

[0075] At block 806, a perception engine 324 of the autonomous vehicle 106 provides the perception data as input to a perception graph, and at block 808, the perception engine 324 uses the perception graph to process the perception data and generate a perception command. The perception command may include any type of result of perception processing of the perception data, including but not limited to an above-ground-level (AGL) value that represents a distance between the autonomous vehicle 302 and the ground, a semantic nudge value (i.e., a value obtained based on semantic labels applied to visible objects that is usable to augment a position of the autonomous vehicle 302 obtained from sources such as a global navigation satellite system), a semantic obstacle detection value that represents obstacles detected based on the perception data, a geofiducial location value that represents a location of a geofi ducial object detected within the perception data (i.e., a uniquely identifiable object associated with a known geographic position), a landing pad location value that represents a location of a landing pad detected within the perception data, or any other type of perceived value. In some embodiments, the perception command includes an indication of data types of the perception data that was used to generate the perception 204049-P103WOcommand. For example, a semantic nudge value may be created based on an intensity' image and telemetry data, while an above-ground-level value may be created based on telemetry data and a depth image. In some embodiments, the perception command includes information that allows the path through the perception graph to be determined. In some embodiments, in addition to storing a value identifying the perception command type, the perception command also includes timestamps of each frame of each type of perception data that was used to generate the perception command, as well as a timestamp at which the perception command itself was created.

[0076] In some embodiments, providing the perception data to the perception graph may constitute providing a pointer to the stream to one or more input nodes / vertices of the perception graph. In some embodiments, the raw stream may be consumed by the input node / vertex. In other embodiments, the input node / vertex may subsample the stream based on the expected processing time for each frame. For example, if a given node / vertex takes 300 ms to run, a frame rate of 30 Hz would quickly cause the node / vertex to get behind as input queues fill faster than work can be completed. In such a case, the input stream may be subsampled to an appropriate rate (e g., 2 Hz) for the node / vertex.

[0077] Ideally, the perception data streams will provide frames at the expected rate and nodes / vertices will complete well within their expected time. However, input queues may fill up for a variety of reasons, including but not limited to unexpected processor load, failures of downstream components, or other reasons. In some embodiments, a graph-wide maximum input queue size may be configured, and the input nodes / vertices of the perception graph may block on Add calls when the maximum input queue size is reached. This generally causes frames to be dropped at the driver level. While this would be reproducible in log replay since such frames would not be received by the input nodes / vertices and would therefore not be logged, some embodiments may handle this situation in a more graceful manner. For example, a FlowLimiterCalculator, designed by MediaPipe, may be used instead of simply dropping packets at the driver level. The FlowLimiterCalculator sits in front of a pipeline of work and has a back connection at the end of that pipeline. Assuming a one-to-one mapping of inputs to outputs (i.e., for each input timestamp, there is exactly one matching output timestamp), it can count the number of packets currently "in-flight" (or in the processing pipeline). Then, FlowLimiterCalculator can drop input frames if a configurable "max in-flight" parameter is exceeded. This effectively serves as a firewall 214049-P103WObetween different nodes / vertices. There is also an "allow" stream that keeps track of which timestamps are dropped. This allows the dropped frames to be logged, and the frame dropping behavior can be reproduced during log replay by replacing the FlowLimiterCalculator with a GateCalculator that uses the "allow" stream as input.

[0078] At block 810, the perception engine 324 stores the perception command in the log data store 316. Because the perception command includes the timestamps of the frames of input perception data that were used to generate the perception command, as well as a timestamp at which the perception command was generated and the types of input perception data that was used to create it, the stored perception command can be used both in log replay and to help generate accurate predictions of computation delay within the perception graph processing, as discussed in further detail below.

[0079] At block 812, the perception engine 324 provides the perception command to a route traversal engine 320. In some embodiments, the perception command is provided to a command queue that is used to queue commands to the route traversal engine 320, and the route traversal engine 320 consumes commands from the queue as appropriate. The route traversal engine 320 is configured to control the components of the autonomous vehicle 302 to cause the autonomous vehicle 302 to autonomously traverse a route (e.g., a flight path, etc.). The perception commands consumed by the route traversal engine 320 help the route traversal engine 320 determine control decisions to successfully traverse the route. As a non-limiting example, the route traversal engine 320 may plan propulsion commands based on a stored map and a location obtained from a global satellite navigation system, but may update the propulsion commands based on a semantic nudge value in a perception command, or based on a semantic obstacle command in a perception command.

[0080] The set of method steps 840 then proceeds to a decision block 814, where a determination is made regarding whether the method 800 is done generating log data. In some embodiments, the set of method steps 840 may proceed for the duration of a flight or other type of mission. In some embodiments, the set of method steps 840 may proceed for a predetermined period of time. In some embodiments, any other suitable period may be used for the set of method steps 840.

[0081] If the determination is made that the method 800 is not done generating log data, then the result of decision block 814 is NO. and the method 800 returns to block 802 to generate further log data. Otherwise, if it is determined that the method 224049-P103WO800 is done generating log data, then the result of decision block 814 is YES, and the method 800 advances from the set of method steps 840 to block 816.

[0082] At block 816, a log collection engine 410 of a simulation computing system 102 receives the perception data and perception commands from the log data store 316 of the autonomous vehicle 106 (or vehicles) and stores the perception data and perception commands in a log data store 408 of the simulation computing system 102. In some embodiments, the log collection engine 410 may receive the logged information via a wired or wireless network from the autonomous vehicle 302. In some embodiments, a removable computer-readable medium may be transferred from the autonomous vehicle 302 to the simulation computing system 102 for the log collection engine 410 to collect the logged information. The method 800 then proceeds to a continuation terminal ("terminal A").

[0083] From terminal A (FIG. 8B), the method 800 proceeds to block 818, where a delay determination engine 414 of the simulation computing system 102 determines predicted delays for combinations of perception data types and perception commands based on the data stored in the log data store 408, and at block 820, the delay determination engine 414 stores the predicted delays in a delay data store 412 of the simulation computing system 102. The predicted delays are used by the simulation computing system 102 when generating perception commands to inject a realistic amount of delay. Since the computational power of the simulation computing system 102 is likely to be faster or slower than the autonomous vehicle 302 that generated the log data, the injection of predicted delays is desirable to increase the fidelity of the simulation.

[0084] In some embodiments, the predicted delays may be calculated by¬ averaging the delays experienced in matching perception commands. As discussed above with regard to block 808, the logged perception commands include a perception command type, a timestamp at which the perception command was generated, and timestamps for the frames of each type of perception data used to create the perception command. Accordingly, the logged perception commands may be used as delay prediction information. To determine matching perception commands, the logged perception commands may be searched for perception commands that have the same perception command type and have the same types of perception data used to create the perception command, thus indicating matching paths through the perception graph. Once the matching perception commands are retrieved from the log data store 408, the 234049-P103WOpredicted delay may be determined by averaging the time between the timestamps of the input perception data and the timestamp of the perception command. The predicted delay may then be stored in the delay data store 412 in association with the perception command type and the type(s) of perception data used to create the perception command, to enable retrieval based on these criteria.

[0085] In some embodiments, other sources of delay prediction information may be used. For example, in testing the perception graph, a developer may wish to replace nodes / vertices with new code to be tested. In such cases, the developer may themselves determine a predicted delay for paths through the perception graph that pass through the new code, either manually or through separate testing, and may provide the determined predicted delay in the delay data store 412.

[0086] At block 822, a perception engine 418 of the simulation computing system 102 obtains perception data. As discussed above, one benefit of the method 800 is that only minimal changes need to be made when operating either in a log replay mode or a simulation mode. Accordingly, at block 822, the perception engine 418 may obtain perception data from the log data store 408 if operating in log replay mode, or may obtain perception data from the simulation engine 416 if operating in simulation mode. One will note that, if the log data includes frames that were processed by the inflight perception graph but not frames that were dropped, then replaying the log data will operate in a deterministic manner with regard to the otherwise non-deterministic frame dropping.

[0087] At block 824, the perception engine 418 provides the perception data as input to a simulation perception graph to generate a perception command, wherein the perception command includes a perception command t pe and an indication of one or more types of perception data used. As with the in-flight perception graph used in the set of method steps 840, the perception engine 418 may provide the perception data as streams to input nodes / vertices of the simulation perception graph. If using logged perception data, the perception engine 418 may load the logged data into the streams input into the simulation perception graph. If using simulated perception data, then the perception engine 418 may connect simulated stream outputs to the input nodes / vertices of the simulation perception graph.

[0088] In some embodiments, the processing performed by the simulation perception graph based on the types of perception data received for a given timestamp may determine the path through the simulation perception graph that is traversed and 244049-P103WOthe perception command type that is generated. In some embodiments, a given input frame with a given timestamp may contribute to the generation of more than one perception command. For example, a given frame of telemetry data may contribute to both an above-ground-level perception command and a semantic nudge perception command, according to the processing performed within the simulation perception graph.

[0089] In some embodiments, instead of merely providing perception data to input nodes / vertices of the simulation perception graph, the perception engine 418 may instead or also provide logged intermediate results to appropriate nodes / vertices of the simulation perception graph. For example (and as illustrated in FIG. 6), the perception engine 418 may provide a logged semantic image 620 as input to a semantic abort 622 node and / or a semantic nudge 624, instead of a generated semantic image 628 generated by an earlier portion of the simulation perception graph. This allows for a variabledepth log replay, and may save processing time (by avoiding simulated processing of earlier portions of the simulation perception graph) and / or storage space (by avoiding storage of the full copy of the input perception data). Because the logged semantic image 620 will include a timestamp of the input data used to create the logged semantic image 620, the eventual perception command created by the simulation perception graph will contain the same timestamp information in a variable-depth log replay as with a full replay.

[0090] At block 826, the perception engine 418 retrieves a predicted delay from the delay data store 412 based on the perception command type and the one or more ty pes of perception data used, and at block 828, the perception engine 418 updates the perception command based on the predicted delay. Any suitable technique may be used to inject the delay into the perception command. Typically, a timestamp of the perception command will be adjusted by adding the predicted delay to the timestamps of the perception data used to create the perception command. Since the predicted delay represents an expected computation time on the autonomous vehicle 302 for the path that the types of perception data took through the perception graph to generate a command of the perception command type, injecting the delay here will cause the perception command to have a timestamp that closely resembles what the timestamp would be if the perception graph was executed using the autonomous vehicle 302, even if the simulation computing system 102 has very different computing power than the autonomous vehicle 302.254049-P103WO

[0091] At block 830, the perception engine 418 provides the perception command to a simulation queue 510, and at block 832, a simulation engine 416 of the simulation computing system 102 processes the perception command in the simulation queue. In some embodiments, the simulation queue orders the perception commands by timestamp, and the simulation engine 416 consumes the perception commands when its own timestamp reaches the timestamp of the next perception command in the simulation queue 510. In some embodiments, the simulation engine 416 and the perception engine 418 may perform in lockstep, and the simulation engine 416 may wait for a perception command to be generated by the perception engine 418 and inserted into the simulation queue 510 before stepping through its next state. Processing of the perception command by the simulation engine 416 may include performing similar changes to control commands for components of the autonomous vehicle 302 as during an actual mission for the autonomous vehicle 302, with simulated performance of the autonomous vehicle 302 by the vehicle simulation 504 in response to the commands following. In some embodiments, the simulation engine 416 may also perform a perception simulation 502 based on the vehicle simulation 504. and feed the simulated perception data back to the perception engine 418 for a subsequent timestamp.

[0092] The method 800 then proceeds to a decision block 834, where a determination is made regarding whether the log replay or simulation is completed. If it is determined that the log replay or simulation is not complete (e g., the simulated mission is not complete, or the entirety of the log has not yet been replayed), then the result of decision block 834 is NO, and the method 800 returns to block 822 to process a subsequent set of perception data. Otherwise, if the log replay or simulation is complete, then the result of decision block 834 is YES, and the method 800 proceeds to an end block and terminates.

[0093] In the preceding description, numerous specific details are set forth to provide a thorough understanding of various embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.264049-P103WO

[0094] Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0095] The order in which some or all of the blocks appear in each method flowchart should not be deemed limiting. Rather, one of ordinary' skill in the art having the benefit of the present disclosure will understand that actions associated with some of the blocks may be executed in a variety of orders not illustrated, or even in parallel.

[0096] The processes explained above are described in terms of computer software and hardware. The techniques described may constitute machine-executable instructions embodied within a tangible or non-transitory machine (e g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, the processes may be embodied within hardware, such as an application specific integrated circuit (“ASIC”) or otherwise.

[0097] The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments of. and examples for, the invention are described herein for illustrative purposes, various modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.

[0098] These modifications can be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.274049-P103WO

Claims

CLAIMSWhat is claimed is:

1. A non-transitory computer-readable medium having computer-executable instructions stored thereon that, in response to execution by one or more processors of a simulation computing system, cause the simulation computing system to perform actions for generating perception commands, the actions comprising:determining, by the simulation computing system, delay prediction information for a perception system, wherein the delay prediction information is usable to predict an amount of delay for an autonomous vehicle computing system to generate perception commands using an in-flight perception graph;obtaining, by the simulation computing system, perception data; providing, by the simulation computing system, the perception data to a simulation perception graph to generate a perception command, wherein the perception command includes a timestamp and an indication of one or more ty pes of perception data used by the simulation perception graph to generate the command;determining, by the simulation computing system, a predicted delay for the perception command based on the delay prediction information;updating, by the simulation computing system, the timestamp of the perception command based on the predicted delay; andproviding, by the simulation computing system, the perception command and the predicted delay to a simulation queue.

2. The non-transitory computer-readable medium of claim 1, wherein the perception data includes two or more of data from an intensify camera, data from a depth camera, data from a Tango camera, or telemetry data.

3. The non-transitory computer-readable medium of claim 1, wherein the perception command includes an above-ground-level value, a semantic obstacle detection value, a geofiducial location value, or a landing pad location value.

4. The non-transitory computer-readable medium of claim 1, wherein determining the delay prediction information includes:284049-P103WOreceiving, by the simulation computing system, log data generated by one or more autonomous vehicle computing systems, wherein the log data includes captured perception data and generated perception commands.

5. The non-transitory computer-readable medium of claim 4, wherein the inflight perception graph and the simulation perception graph have matching nodes and edges.

6. The non-transitory computer-readable medium of claim 5, wherein determining the delay prediction information includes:analyzing, by the simulation computing system, the log data to determine average processing times for combinations of perception commands and perception data types.

7. The non-transitory computer-readable medium of claim 4. wherein the inflight perception graph and the simulation perception graph have at least one node that is different.

8. The non-transitory computer-readable medium of claim 7, wherein determining the delay prediction information includes:receiving, by the simulation computing system from a developer, a predicted delay related to the at least one node that is different.

9. The non-transitory computer-readable medium of claim 1, wherein obtaining the perception data includes retrieving at least a portion of the perception data from a log received from an autonomous vehicle computing system.

10. The non-transitory computer-readable medium of claim 1, wherein obtaining the perception data includes generating at least a portion of the perception data using a simulation.

11. A computer-implemented method of generating perception commands, the method comprising:determining, by a simulation computing sy stem, delay prediction information for a perception system, wherein the delay prediction information is usable to predict294049-P103WOan amount of delay for an autonomous vehicle computing system to generate perception commands using an in-flight perception graph;obtaining, by the simulation computing system, perception data; providing, by the simulation computing system, the perception data to a simulation perception graph to generate a perception command, wherein the perception command includes a timestamp and an indication of one or more t pes of perception data used by the simulation perception graph to generate the command;determining, by the simulation computing system, a predicted delay for the perception command based on the delay prediction information;updating, by the simulation computing system, the timestamp of the perception command based on the predicted delay; andproviding, by the simulation computing system, the perception command and the predicted delay to a simulation queue.

12. The computer-implemented method of claim 11. wherein the perception data includes two or more of data from an intensity camera, data from a depth camera, data from a Tango camera, or telemetry data.

13. The computer-implemented method of claim 11, wherein the perception command includes an above-ground-level value, a semantic obstacle detection value, a geofiducial location value, or a landing pad location value.

14. The computer-implemented method of claim 11. wherein determining the delay prediction information includes:receiving, by the simulation computing system, log data generated by one or more autonomous vehicle computing systems, wherein the log data includes captured perception data and generated perception commands.

15. The computer-implemented method of claim 14, wherein the in-flight perception graph and the simulation perception graph have matching nodes and edges.

16. The computer-implemented method of claim 15. wherein determining the delay prediction information includes:analyzing, by the simulation computing system, the log data to determine average processing times for combinations of perception commands and perception data types.304049-P103WO17. The computer-implemented method of claim 14, wherein the in-flight perception graph and the simulation perception graph have at least one node that is different.

18. The computer-implemented method of claim 17, wherein determining the delay prediction information includes:receiving, by the simulation computing system from a developer, a predicted delay related to the at least one node that is different.

19. The computer-implemented method of claim 11. wherein obtaining the perception data includes retneving at least a portion of the perception data from a log received from an autonomous vehicle computing system.

20. The computer-implemented method of claim 11. wherein obtaining the perception data includes generating at least a portion of the perception data using a simulation.314049-P103WO

Images