Methods, devices, and systems for evaluating counter-unmanned aircraft systems (c-UAS)
The programmatic method and RTI TruFlight™ platform address the inadequacies of current C-UAS evaluation by providing standardized, efficient, and objective analysis of UAS performance data, enhancing reliability and operational readiness.
Patent Information
- Application Number
- PCT/US2026/014302
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-02-07
- Filing Date
- 2026-02-06
- Publication Date
- 2026-11-12
AI Technical Summary
Current methods for evaluating counter-unmanned aircraft systems (C-UAS) are inadequate, often time-intensive, prone to human error, and produce biased or incomplete results, lacking robust validation, which undermines the effectiveness and operational reliability of these systems.
A programmatic method and computing system for receiving and analyzing first truth data and detection data from UAS test flights, determining performance data, and storing it for evaluation, utilizing a platform like RTI TruFlight™ that supports standardized experimental design and reporting, enabling near-real-time comparison and normalization of data across different C-UAS technologies.
Provides reliable, objective, and efficient evaluation of C-UAS systems, reducing manual analysis time, minimizing errors, and ensuring consistent and actionable insights for law enforcement agencies, facilitating better decision-making and technology deployment.
Smart Images

Figure US2026014302_12112026_PF_FP_ABST
Abstract
Description
Docket No: 395 / 48 PCTTITLEMETHODS, DEVICES, AND SYSTEMS FOR EVALUATING COUNTER-UNMANNED AIRCRAFT SYSTEMS (C-UAS)CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 755.690 titled “METHODS. DEVICES, AND SYSTEMS FOR EVALUATING COUNTER-UNMANNED AIRCRAFT SYSTEMS (C-UAS),” filed on February 7, 2025, which is incorporated herein by its entirety by this reference.TECHNICAL FIELD
[0002] The present invention relates generally to analysis of aviation data and associated positional data, and more specifically to evaluation of counter-unmanned aircraft systems (C-UAS).BACKGROUND
[0003] Unmanned aircraft systems (UASs), unmanned aerial vehicles (UAVs), and drones (hereto referred to as UAS for consistency) are increasingly being exploited for illicit activities. This makes their reliable detection and mitigation an urgent priority for the authorities. However, several challenges complicate this effort. The wide variability in drone designs, inconsistent communication protocols, and the prevalence of radio frequency (RF) noise significantly hinder detection capabilities. Environmental factors, such as adverse weather conditions, physical obstructions, and radar clutter, further intensify these difficulties. Moreover, locating drone operators, who often remain hidden or mobile, adds another layer of complexity. Despite the growing demand for counter-unmanned aircraft systems (C-UASs), advancements in testing and evaluation of these systems often fail to keep pace.Docket No: 395 / 48 PCT
[0004] Current approaches are frequently inadequate. Manual analysis can be both time-intensive and prone to human error, while vendor-driven tools sometimes produce biased or incomplete results. As a result, agencies are left to rely on systems that lack robust validation, undermining their confidence in effectiveness and operational reliability of these technologies. Addressing these gaps is critical to developing comprehensive and trusted C-UAS solutions. As such, new methods, devices, and systems are needed for C-UAS evaluation.SUMMARY
[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
[0006] Disclosed herein are methods, devices, and systems for evaluation of counterunmanned aircraft systems (C-UAS). In one embodiment, a programmatic method includes receiving first truth data associated with a first test flight of a first UAS and receiving first detection data associated with the first test flight. The first detection data is provided by a first C-UAS monitoring the first test flight. The programmatic method further includes determining first performance data of the first C-UAS based on the first truth data and the first detection data and storing the first performance data in a memory.
[0007] In another embodiment, a computing system having a processor and a memory is disclosed. The computing system is configured for receiving first truth data associated with a first test flight of a first UAS and receiving first detection data associated with the first test flight. The first detection data is provided by a first C-UAS monitoring the first test flight. The computing system is further configured for determining first performance data of the first C-UAS based on the first truth data and the first detection data and storing the first performance data in the memory.
[0008] In another embodiment, a non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium stores instructions to beDocket No: 395 / 48 PCTimplemented on at least one computing device including at least one processor, the instructions when executed by the at least one processor cause the at least one computing device to perform a method. The method includes receiving first truth data associated with a first test flight of a first UAS and receiving first detection data associated with the first test flight. The first detection data is provided by a first C-UAS monitoring the first test flight. The method further includes determining first performance data of the first C-UAS based on the first truth data and the first detection data and storing the first performance data in a memory.
[0009] In another embodiment, a method for C-UAS evaluation is disclosed. The method includes conducting a first test flight of a first UAS. The method further includes recording first truth data associated with the first test flight and recording first detection data associated with the first test flight. The first detection data is provided by a first C-UAS monitoring the first test flight. The method further includes providing the first truth data and the first detection data to a computing system. The computing system is configured for determining first performance data of the first C-UAS based on the first truth data and the first detection data and storing the first performance data in a memory.BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The present embodiments are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings. In the drawings:
[0011] FIG. 1 depicts a flow chart illustrating a programmatic method for counterunmanned aircraft systems (C-UAS) evaluation in accordance with embodiments of the present disclosure.
[0012] FIG. 2 depicts a flow chart illustrating a method for C-UAS evaluation by a C-UAS test facility in accordance with embodiments of the present disclosure.Docket No: 395 / 48 PCT
[0013] FIG. 3 depicts a block diagram illustrating a system for providing one or more of the previously described methods discussed with FIG. 1 and FIG. 2 in accordance with embodiments of the present disclosure.
[0014] FIG. 4 depicts a block diagram illustrating one embodiment for a system server of the system described in FIG. 3 in accordance with embodiments of the present disclosure.
[0015] FIG. 5 depicts a block diagram illustrating one embodiment of a workstation of the system described in FIG. 3 in accordance with embodiments of the present disclosure.
[0016] FIG. 6 depicts a diagram illustrating one embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.
[0017] FIG. 7 depicts a diagram illustrating another embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.
[0018] FIG. 8 depicts a diagram illustrating another embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.
[0019] FIG. 9 depicts a diagram illustrating another embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.
[0020] FIG. 10 depicts a diagram illustrating another embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.
[0021] FIG. 11 depicts a diagram illustrating another embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.
[0022] FIG. 12 depicts a diagram illustrating another embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.
[0023] FIG. 13 depicts a diagram illustrating another embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.Docket No: 395 / 48 PCT
[0024] FIG. 14 depicts a diagram illustrating another embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.
[0025] FIG. 15 depicts a diagram illustrating another embodiment of a GUI provided by the system of FIG. 3 in accordance with embodiments of the present disclosure.DETAILED DESCRIPTION
[0026] The presently disclosed subject matter is described with specificity to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventors have contemplated that the claimed invention might also be embodied in other ways, to include different steps or elements similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the term “step” may be used herein to connote different aspects of methods employed, the term should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.
[0027] To date, UASs have presented a novel and increasingly prevalent mechanism for illicit activity, including terrorist attacks, drug and weapons trafficking, and nefarious surveillance. Detecting a UAS of concern is a complex and multifaceted challenge due to several issues. A first issue is that diversity in UAS design including variability in drone types, sizes (ranging from microdrones to larger UASs), and their low / high radar cross-sections (RCS) complicates detection efforts. Due to this diversity and a rapidly evolving drone landscape, the ability to keep up with interpreting, understanding, and processing related data at all or within a timely manner remains difficult. A second issue is that complex communication protocols including the use of encrypted, frequency-hopping, or non-standard communication links makes it difficult to intercept or identify drone signals. A third issue involves limitations of single-sensor modalities. No single sensing technology such as radar, radio frequency (RF), acoustic, or optical systems can reliably detect all drone types under all conditions. A fourth issue involves cluttered RF environments. In dense urban areas, distinguishing drone control signals from other RF noiseDocket No: 395 / 48 PCT(e.g., Wi-Fi, Bluetooth®, or cellular networks) is highly challenging. A fifth issue involves radar clutter and environmental factors, false positives caused by birds, moving vehicles, and environmental challenges such as rain, fog, and dense foliage reduce radar effectiveness. A sixth issue involves drone operator detection. Beyond the drone itself, detecting and locating the operator controlling the drone is a critical and complex task, particularly in urban or crowded areas.
[0028] Disclosed herein are methods, devices, and systems for comprehensive C-UAS evaluation to address the emerging threats posed by unauthorized drone usage while overcoming the above identified issues. The outputs of these methods devices and systems produce actionable reports for the appropriate law enforcement agencies.
[0029] FIG. 1 depicts a flow chart 100 illustrating one embodiment of a programmatic method for C-UAS evaluation in accordance with embodiments of the present disclosure. The programmatic method may be implemented on a laptop, a personal computer, a workstation, a smart tablet, a smart phone, or the like. The programmatic method may also be implemented as a progressive web application (PWA). In other embodiments, the programmatic method may be implemented on a server or a plurality of servers. In further embodiments, the server or plurality of servers may be a portion of a networked computing environment. For example, the networked computing environment may be a cloud computing environment. In certain embodiments, the programmatic method may be implemented on a virtual server or a plurality of virtual servers. In certain embodiments, the server may include an operating system (OS) hosted within a container running on the server and the programmatic method may be executed over the OS hosted within the container. In further embodiments, the container may be a Docker® container.
[0030] In step 102, the programmatic method includes receiving first truth data associated with a first test flight of a first UAS. First truth data may refer to ground truth data collected by a UAS, which serves as the "correct" or verified information about a specific area or features captured in flight, used to calibrate and validate the accuracy of the UAS’s sensor data by comparing it to physical measurements taken on the ground (e.g., C-UAS sensors). The first truth data may be provided in near real time or retrieved from stored files and / or databases. In someDocket No: 395 / 48 PCTembodiments, the first truth data may be derived from at least one test card used for defining the first test flight.
[0031] In step 104, the programmatic method further includes receiving first detection data associated with the first test flight. The first detection data is provided by a first C-UAS monitoring the first test flight. The first detection data may be provided in near real time or retrieved from stored files and / or databases.
[0032] In step 106, the programmatic method further includes determining first performance data of the first C-UAS based on the first truth data and the first detection data.
[0033] In step 108, the programmatic method further includes storing the first performance data in a memory for the C-UAS evaluation. In further embodiments, the programmatic method may further include storing the first performance data as a spreadsheet compatible file.
[0034] In some embodiments, the first truth data may include first test dynamic UAS data associated with the first test flight. In some embodiments, the first test dynamic UAS data may be provided by internal sensors of the UAS. In other embodiments, the first test dynamic UAS data may be provided by external sensors affixed to the first UAS for monitoring the first UAS during the first test flight. In still other embodiments, a first portion of the first test dynamic UAS data may be provided by internal sensors of the UAS and a second portion of the first test dynamic UAS data may be provided by external sensors affixed to the first UAS for monitoring the first UAS during the first test flight.
[0035] In some embodiments, the first test dynamic UAS data may include first location data. The first location data may be provided by a global positioning system (GPS) receiver embedded on the first UAS or attached external to the first UAS. In some embodiments, the first test dynamic UAS data may also include pitch data, yaw data, roll data, vibration data linear acceleration data, velocity data, altitude data, rate of climb data and / or rate of descent data.
[0036] In some embodiments, the first truth data may also include weather dataDocket No: 395 / 48 PCTassociated with the first test flight, hi further embodiments, the weather data may be received from a weather server. In certain embodiments, the weather data, may be received from a National Weather Service application programming interface (API), an OpenWeatherMap API, a Weatherbit API, an AccuWeather API, a Dark Sky API, a Weather2020 API, a ClimaCell API, a Visual Crossing API, or a AerisWeather API.
[0037] In some embodiments, the weather data may be received from weather sensors local to the first test flight. The weather sensors may be on the first UAS, another airborne device, and / or on the ground at a test facility associated with the first test flight.
[0038] In some embodiments, the first truth data may include first static UAS data associated with the first UAS. In further embodiments, the first static UAS data may include a first model number of the first UAS. In still further embodiments, the first static UAS data may include first published performance data associated with the first UAS. Additionally, the first static UAS data may include prior performance data collected from a previous test flight.
[0039] In some embodiments, the first truth data may be received in a spreadsheet format. In further embodiments, the spreadsheet format may be a comma- separated values (CSV) American Standard Code for Information Interchange (ASCII) format. In certain embodiments, the spreadsheet format may be a Microsoft Excel® format. In some embodiments, the first detection data may also be received in a spreadsheet format. In further embodiments, the spreadsheet format may be a CSV ASCII format. In certain embodiments, the spreadsheet format may also be a Microsoft Excel® format.
[0040] In some embodiments (not shown in FIG. 1 ), the programmatic method may further include normalizing the first truth data and normalizing the first detection data. Additionally determining first performance data of the first C-UAS may be further based on the normalized first truth data and the normalized first detection data.
[0041] In some embodiments, the programmatic method may further include generating test card data based on the normalized first truth data and the normalized first detectionDocket No: 395 / 48 PCTdata and storing the test card data in the memory. In further embodiments, the programmatic method may further include providing at least a portion of the test card data to a graphical user interface (GUI). In still further embodiments, the programmatic method may further include receiving user updated test card data from the GUI and storing the user updated test card data in the memory.
[0042] In some embodiments, the programmatic method may further include updating test card data based on the normalized first truth data and the normalized first detection data, and storing the program updated test card data in the memory. In further embodiments, the programmatic method may further include providing at least a portion of the program updated test card data to a GUI. In still further embodiments, the programmatic method may further include receiving user updated test card data from the GUI and storing the user updated test card data in the memory.
[0043] In some embodiments, the programmatic method may further include generating multi-waypoint test data based on the normalized first truth data and the normalized first detection data and providing the multi-waypoint test data to a GUI. In further embodiments, the programmatic method may further include receiving updated multi-waypoint test data from the GUI and storing the updated multi-waypoint test data in the memory.
[0044] In some embodiments, the first truth data and the first detection data may be received in near real time during the first test flight.
[0045] In some embodiments, the programmatic method may further include providing the first performance data to a GUI. In further embodiments, the GUI may be provided by a smart phone, a smart tablet, a laptop, a personal computer, a workstation, or the like.
[0046] In some embodiments the first truth data and the first detection data may be received in near real time during the first test flight and the first performance data may also be provided to the GUI in near real time during the first test flight.
[0047] In some embodiments (not shown in FIG. 1), the programmatic method mayDocket No: 395 / 48 PCTfurther include receiving second truth data associated with a plurality of test flights associated with a plurality of UASs and receiving second detection data associated with the plurality of test flights. In certain embodiments, the second truth data may be derived from one or more test cards used to define the plurality of test flights. Additionally, the second truth data may be received from one or more sensors associated with the plurality of UASs. The second detection data may be provided by a plurality of C-UASs monitoring the plurality of test flights. The programmatic method may further include determining second performance data of the plurality of C-UASs based on the second truth data and the second detection data and storing the second performance data in a memory. In further embodiments, the programmatic method may further include providing the first performance data and the second performance data to a GUI in a comparative manner between the C-UASs. In further embodiments, the GUI may be provided by a smart phone, a smart tablet, a laptop, a personal computer, a workstation, or the like.
[0048] In some embodiments (not shown in FIG. 1), the programmatic method may further include (1) normalizing the first truth data and normalizing the first detection data, wherein determining first performance data of the first C-UAS is further based on the normalized first truth data and the normalized first detection data; (2) normalizing the second truth data and normalizing the second detection data, wherein determining second performance data of the plurality of C-UASs is further based on the normalized second truth data and the normalized second detection data; and (3) providing the first performance data and the second performance data to a GUI.
[0049] In some embodiments, the programmatic method may further include generating test card data based on the normalized first truth data, the normalized first detection data, the normalized second truth data, and the normalized second detection data; and storing the test card data in the memory. In further embodiments, the programmatic method may further include providing at least a portion of the test card data to a GUI. In still further embodiments, the programmatic method may further include receiving user updated test card data from the GUI and storing the user updated test card data in the memory.
[0050] In some embodiments, the programmatic method may further include updatingDocket No: 395 / 48 PCTtest card data based on the normalized first truth data, the normalized first detection data, the normalized second truth data, and the normalized second detection data; and storing the program updated test card data in the memory. In further embodiments, the programmatic method may further include providing at least a portion of the program updated test card data to a GUI. In still further embodiments, the programmatic method may further include receiving user updated test card data from the GUI and storing the user updated test card data in the memory.
[0051] In some embodiments, the programmatic method may further include generating multi-waypoint test data based on the normalized first truth data, the normalized first detection data, the normalized second truth data, and the normalized second detection data; and providing the multi-waypoint test data to a GUI. In further embodiments, the programmatic method may further include receiving updated multi- waypoint test data based from the GUI and storing the updated multi-waypoint test data in the memory.
[0052] In some embodiments, the programmatic method may be implemented within a spreadsheet application. The spreadsheet application may be a Microsoft Excel® spreadsheet application. The spreadsheet application may be configured to provide the first performance data and / or the second performance data in a table format, a chart format, a dashboard format, and / or the like.
[0053] FIG. 2 depicts a flow chart 200 illustrating another embodiment of a method for C-UAS evaluation by a C-UAS test facility in accordance with embodiments of the present disclosure. In step 202, the method includes conducting a first test flight of a first UAS.
[0054] In step 204, the method further includes recording first truth data associated with the first test flight.
[0055] In step 206, the method further includes recording first detection data associated with the first test flight. The first detection data is provided by a first C-UAS monitoring the first test flight.
[0056] In step 208, the method further includes providing the first truth data and theDocket No: 395 / 48 PCTfirst detection data to a computing system. The computing system is configured for determining first performance data of the first C-UAS based on the first truth data and the first detection data and storing the first performance data in a memory as disclosed in the discussion of FIG. 1.
[0057] The method (not shown in FIG. 2) may further include (1) conducting a plurality of test flights with a plurality of UASs; (2) recording second truth data associated with the plurality of test flights; (3) recording second detection data associated with the plurality of test flights, wherein the second detection data is provided by a plurality of C-UASs monitoring the plurality of test flights; (4) providing the second truth data and the second detection data to the computing system, wherein the computing system is further configured for determining second performance data on the plurality of C-UASs; and (5) storing the second performance data in the memory.
[0058] FIG. 3 depicts a block diagram illustrating a system for providing one or more of the previously described methods discussed with FIG. 1 and FIG. 2 in accordance with embodiments of the present disclosure. The system 300 includes a system server 302 executing a system server application 304. The system server application 304 may implement any of the previously described methods disclosed herein. The system 300 also includes a workstation 306 executing a workstation application 308. The workstation application 308 may implement any of the previously described methods disclosed herein. The system server 302 is configured to communicate with the workstation 306 over the network 310. In some embodiments, the system server application 304 will execute a first portion of the methods described herein and the workstation application 308 will execute a second portion of the methods described herein. The network 310 may be any type or combination of wired, wireless, and / or optical networks. The network 310 may include one or more wide area networks (WANs), local area networks (LANs), and personal area networks (PANs). The network 310 may also include the Internet. The system 300 also includes a C-UAS evaluation facility 312 for performance testing and / or certification of C-UASs. The C-UAS evaluation facility 312 may provide truth data and detection data as discussed in FIG 1 in near real time over the network 310.Docket No: 395 / 48 PCT
[0059] FIG. 4 depicts one embodiment for the system server 302 of the system 300 of FIG. 3 in accordance with embodiments of the present disclosure. The system server 302 is a hardware server and may include a processor 402, a main memory 404, a database 406, a datacenter network interface 408, and an administration user interface (UI) 410. The system server 302 may be configured to host a virtual server. For example, the virtual server may be an Ubuntu® server, RedHat® (i.e. Linux) server, OpenBSD® server, FreeBSD® server, or the like. The system server 302 may also be configured to host a virtual container. For example, the virtual container may be the Docker® virtual container or the like. In some embodiments, the virtual server and or virtual container may be distributed over a plurality of hardware servers using hypervisor technology. The system server 302 may be implemented in the Microsoft Azure®, the Amazon Web Services® (AWS). or the like cloud computing data center environments. In other embodiments, the system server 302 may be hosted within a business or residential environment.
[0060] The processor 402 may be a multi-core server class processor suitable for hardware virtualization. The processor 402 may support at least a 64-bit architecture and a single instruction multiple data (SIMD) instruction set. The main memory 404 may include a combination of volatile memory (e.g., random access memory) and non-volatile memory (e.g., flash memory). The database 406 may include one or more hard drives. In some embodiments, the database 306 may be an open-source database such as the MongoDB® database, the PostgreSQL® database, or the like. The database 406 may be configured to store one or more ASIC libraries and one or more ASIC designs.
[0061] The enterprise network interface 408 may provide one or more high-speed communication ports to the data center switches, routers, and / or network storage appliances. The enterprise network interface 408 may include high-speed optical Ethernet, InfiniBand (IB), Internet Small Computer System Interface iSCSI, and / or Fibre Channel interfaces. The administration UI 410 may support local and / or remote configuration of the system server 302 by a data center administrator or the like. The enterprise network interface 408 allows the system server 302 to communicate over the network 310.Docket No: 395 / 48 PCT
[0062] FIG. 5 depicts a block diagram 500 illustrating the workstation 306 of the system 300 described in FIG. 3 in accordance with embodiments of the present disclosure. The workstation 306 may be configured to communicate with the system server 302 over the network 310 in a similar manner as described in the system 300 of FIG. 3. The workstation 306 may be configured to host a specific application (e.g., the workstation application 308), a browser application, a third-party application, and / or the like. The workstation 306 includes at least one processor 502, a memory 504, a display 506, a user interface (UI) 508, and a network interface 510.
[0063] The workstation 306 may include an operating system (OS) such as a Windows® OS, a Macintosh® OS, a Linux® OS. or the like. The memory 504 may include a combination of volatile memory (e.g., random access memory) and non-volatile memory (e.g., solid state drive and / or hard drives). The display 506 may be an external display (e.g., computer monitor) or internal display (e.g., laptop). The UI 508 may include a keyboard, and a pointing device (e.g., mouse). The network interface 510 may be a wired Ethernet interface or a Wi-Fi interface. Additionally, the workstation 306 may include one or more interfaces (not shown in FIG. 5) for C-UAS evaluation.
[0064] In a preferred embodiment. Research Triangle Institute’s RTI TruFlight™ platform is disclosed. The platform transforms the way C-UAS s are tested and evaluated. The platform has the capability to ingest many data types (e.g., radar, RF, Drone-ID, UAS truth data, etc.). The platform supports experimental design options (e.g., test card specifications) and reporting features including metrics, statistical analysis, graphs, images, and associated information. The reporting features may also include having the ability to annotate notes and collate a summary of different test flight conditions from aggregate data. In certain embodiments, "test card" is a tool used in experimental design and project management to structure and plan experiments by clearly defining hypotheses to include both independent and dependent variables, outlining how to test them, and specifying the metrics to measure success. The platform allows for a test card to be created by a user in a standardized format prior to the beginning of a testing event.Docket No: 395 / 48 PCTThe test card also facilitates the logging and analysis of C-UAS detection data (and possibly truth data). This capability further facilitates the normalization of the C-UASs’ evaluations.
[0065] The platform transforms the way C-UAS technologies are tested and evaluated. By providing a single user interface, the platform ensures standardized and consistent experimental designs, enabling more reliable and objective evaluations. The platform allows users to compare C-UAS data with UAS truth data in near real-time, streamlining analysis and eliminating the need for manual interpretation. This capability saves significant time and can reduce errors while providing data and performance metrics for determining clear and actionable insights from evaluations of C-UASs. The platform also delivers normalized reporting structures, offering agencies a reliable way to assess the performance of different C-UAS technologies. This ensures that operational decisions are backed by robust, objective data, empowering agencies to deploy effective solutions with confidence.
[0066] Exemplary applications for the platform include (1) federal, state, and local law enforcement agencies requiring reliable drone detection systems; (2) evaluation environments for fusion-type system integrations; and (3) research and development teams testing cutting-edge C-UAS technologies. Advantages include (1) comprehensive data handling including supports for radar, RF, drone- ID, acoustic, remote-ID, and UAS truth data inputs; (2) flexible experimental design including customizable test card specifications; (3) detailed reporting including metrics, statistical analysis, and graphical insights; and (4) efficiency that saves time and reduces the workload of backend data analysis.
[0067] The platform provides a vendor-agnostic data management, reporting, and visualization tool that is paramount to determining the efficacy of a single C-UAS sensor or multimodal sensor approach. As such, the platform supports better adoption choices, enhances investigative capabilities, and reduces the gap between technology advancement and community awareness.Docket No: 395 / 48 PCT
[0068] The platform is suitable for on-premises installations due to the sensitivity of the operations allowing data to be directly downloaded from onsite servers. In other scenarios the platform can be a cloud- based offering as required by the customer or law enforcement agency.
[0069] The platform can receive data exported from C-UAS technologies which varies by sensor and manufacturer. C-UAS raw data is commonly in JSON, BIN, or CSV file format. Additionally, an API may be used for accessing the data. Based on input drone model, the platform can receive UAS telemetry data as part of the truth data. Typically, the UAS telemetry data would be formatted in Apache Parquet and / or binary (i.e. ‘.dat’). The platform can also receive supplemental GPS transponder data exported in a spreadsheet format (e.g., CSV ASCII format). Data scrubbing (i.e. cleansing) may be performed by the platform as necessary for viewing by nonauthorized personnel.
[0070] For accuracy and sensitivity of the C-UAS evaluations and valuations, a library of C-UAS s and UASs with representative test flights is maintained. Internal telemetry data and external GPS transponders are used. Truth data is then compared to the C-UAS detections of that drone in time and space. Benchmarks include the accuracy of the C-UAS detection in time and space (including range, altitude, azimuth), latency of the detection (lag in detection), probability of detection (especially in differing environments / conditions). and the maintained tracking (tracking efficacy) of the UAS.
[0071] In general, the platform is agnostic to the various data streams and is capable of normalizing the various data streams in near real time.
[0072] The platform allows multiple waypoints for the test flights. Additionally, the platform can use standardized test flights and waypoints. There is also a list of tests that are flight patterns between the waypoints.
[0073] The platform solves the problem associated with large amounts of metadata leading to manual trimming and analysis. Typically, this manual trimming can take weeks or months of manual hours to complete.Docket No: 395 / 48 PCT
[0074] The platform also solves the problem of unvalidated capabilities and unverified operational efficacy of C-UAS systems. The platform also solves the issue of reliance on vendor-driven analysis. This vendor-driven analysis is often skewed in favor of the vendor. This also leads to an inability to compare the output of multiple C-UAS technologies, creating an “apples to oranges” analysis for the potential buyer / customer
[0075] The platform provides a single user interface to establish and record evaluation parameters and metrics to ensure consistent and standardized methods of experimental design for C-UAS. Visual comparisons of C-UAS data to UAS truth data may be provided in near real-time if truth and detection data are received in near real time.
[0076] The platform provides a streamlined analysis of complex C-UAS data sets efficiently without manual interpretation. As such, considerable time on back-end analytics is saved. Further a normalized reporting structure to determine the performance efficacy of various C-UAS systems through an objective assessment.
[0077] The platform has many additional features for the facilitation of C-UAS evaluation. For example, the platform is capable of being provided over the Internet with users’ permissions controlled through login with Microsoft Single Sign-On (SSO). The platform further allows the development of standardized testing parameters (flight patterns, common approaches, day / night, environment, RF noise, etc.). The platform is capable of ingesting UAS truth time series data from a CSV format containing at a minimum column including timestamp, latitude, longitude, altitude with respect to sea level or another defined altitude, and an operator location.
[0078] The platform is further capable of ingesting and sanitizing C-UAS detection time series data in CSV, JavaScript Object Notation (JSON), and Parquet formats containing at a minimum column including timestamp, latitude, longitude, altitude with respect to sea level, signal strength and signal bearing, operator location, photograph, and video recording. The platform is also capable of recording parameters around the test flight including type of drone, conditions (e.g„ weather, night or day. etc.), and sensor location.Docket No: 395 / 48 PCT
[0079] As a key feature, the platform is capable of providing a scorecard representing performance of the C-UAS system under test relative to the truth data of the drone and test flight, and relative to other C-UAS tests. The platform is further capable of recording general metadata about each test flight and its associated environmental conditions.
[0080] For detailed analysis, the platform is capable of using potential algorithms for performance including simple multivariate regression, root mean square (RMS) Haversine distance difference, and time- varying granger causality for conditions (e.g.. weather, night or day, etc.), flight path and pattern, and sensor location. The platform is also capable of providing a map interface to overlay both UAS truth data and C-UAS detection data. The map interface may be static or dynamic and may include geographical representations.
[0081] Additionally, the platform is capable of analyzing repeated tests in an automated fashion including snapshots camera feeds based on trigger alerts, adjusting the granularity of scoring, and detection drop-outs of the C-UAS.
[0082] FIG. 6 through FIG. 15 depict a sampling of GUIs for the previously described devices methods and systems in accordance with embodiments of the present disclosure. FIG. 6 depicts a diagram 600 illustrating a GUI 602 providing an interface including a map for managing waypoint locations for a UAS during C-UAS testing. FIG. 7 depicts a diagram 700 illustrating a GUI 702 providing an interface for a test card maker for UAS flights during C-UAS testing. FIG.8 depicts a diagram 800 illustrating a GUI 802 providing an interface for managing C-UAS configurations. FIG. 9 depicts a diagram 900 illustrating a GUI 902 providing new projects creation including C-UAS configurations (and existing projects) using the interface of GUI 802. FIG. 10 depicts a diagram 1000 illustrating a GUI 1002 providing an interface for creating a new test name “complex multi-waypoint test” and related test parameters. FIG. 11 depicts a diagram 1100 illustrating a GUI 1102 providing an interface displaying logs (including test details, recordings, and data logs) associated with the “complex multi-waypoint test.” FIG. 12 depicts a diagram 1200 illustrating a GUI 1202 providing the interface of GUI 1102 for drone selection. FIG. 13 depicts a diagram 1300 illustrating a GUI 1302 providing an interface for the “complexDocket No: 395 / 48 PCTmulti-waypoint test” including aggregate analytics. FIG. 14 depicts a diagram 1400 illustrating a GUI 1402 providing an interface for the “complex multi-waypoint test” including detailed analytics. FIG. 15 depicts a diagram 1500 illustrating a GUI 1502 that provides a drone inventory management interface. Drone types may be entered including manufacture specifications.
[0083] In summary, the C-UAS sensor market is rapidly evolving. However, there is no structured process for vetting solutions for efficacy related to specific operational use cases, specifically those use cases that are outside of vendor-based evaluation. Currently, the vetting requires third party evaluation. As a result, the processes are complicated by large volumes of data in varying formats with no single analytical solution available. The methods, devices, and systems disclosed herein provide solutions for a vendor-agnostic platform for collecting and evaluating C-UAS performance to inform actionable responses. These solutions include comprehensive data collection and evaluation, support for a multimodal sensor approach (i.e. system of systems), processing of data in varying formats, and facilitating direct comparison of C-UASs.
[0084] The platform is well suited to match the upcoming needs of the Department of Justice (DOI), Drug Enforcement Administration (DEA), Department of Homeland Security (DHS), Federal Aviation Administration (FAA), Federal Bureau of Prisons (BOP), Federal Bureau of Investigation (FBI), Department of Defense (DoD), Department of Energy (DOE), state and local Law Enforce Agencies (LEAs), and the Irregular Warfare Technical Support Directorate (IWTSD).
[0085] While the embodiments have been described in connection with the preferred embodiments of the various figures, it is to be understood that other similar embodiments may be used or modifications and additions may be made to the described embodiment for performing the same function without deviating therefrom. Therefore, the disclosed embodiments should not be limited to any single embodiment, but rather should be construed in breadth and scope in accordance with the appended claims.
Claims
Docket No: 395 / 48 PCTCLAIMSWhat is claimed is:
1. A programmatic method comprising:receiving first truth data associated with a first test flight of a first unmanned aircraft system (UAS);receiving first detection data associated with the first test flight, wherein the first detection data was provided by a first counter-unmanned aircraft system (C-UAS) monitoring the first test flight;determining first performance data of the first C-UAS based on the first truth data and the first detection data; andstoring the first performance data in a memory.
2. The programmatic method of claim 1, wherein the first truth data comprises first test dynamic UAS data associated with the first test flight.
3. The programmatic method of claim 2, wherein the first test dynamic UAS data comprises pitch data, yaw data, or roll data.
4. The programmatic method of claim 2, wherein the first test dynamic UAS data comprises first location data.
5. The programmatic method of claim 2, wherein the first test dynamic UAS data comprises vibration data or linear acceleration data.
6. The programmatic method of claim 2, wherein the first test dynamic UAS data comprises velocity data or altitude data.Docket No: 395 / 48 PCT7. The programmatic method of claim 2, wherein the first test dynamic UAS data comprises rate of climb data and rate of descent data.
8. The programmatic method of claim 2, wherein the first test dynamic UAS data is provided by internal sensors of the UAS.
9. The programmatic method of claim 2, wherein the first test dynamic UAS data is provided by external sensors affixed to the first UAS for monitoring the first UAS during the first test flight.
10. The programmatic method of claim 1, wherein the first truth data comprises weather data associated with the first test flight.
11. The programmatic method of claim 10, wherein the weather data is received from a weather server.
12. The programmatic method of claim 10, wherein the weather data, is received from a National Weather Service application programming interface (API), an OpenWeatherMap API, a Weatherbit API, an AccuWeather API, a Dark Sky API, a Weather2020 API, a ClimaCell API, a Visual Crossing API, or a AerisWeather API.
13. The programmatic method of claim 10, wherein the weather data is received from sensors local to the first test flight.
14. The programmatic method of claim 1, wherein the first truth data comprises first static UAS data associated with the first UAS.
15. The programmatic method of claim 14, wherein the first static UAS data includes a first model number of the first UAS.Docket No: 395 / 48 PCT16. The programmatic method of claim 14, wherein the first static UAS data includes first published performance data associated with the first UAS.
17. The programmatic method of claim 1, wherein the first truth data is received in a spreadsheet format.
18. The programmatic method of claim 17, wherein the spreadsheet format is a comma-separated values (CSV) format.
19. The programmatic method of claim 1, wherein the first detection data is received in a spreadsheet format.
20. The programmatic method of claim 19, wherein the spreadsheet format is a comma-separated values (CSV) format.
21. The programmatic method of claim 1 further comprising normalizing the first truth data and normalizing the first detection data, wherein determining first performance data of the first C-UAS is further based on the normalized first truth data and the normalized first detection data.
22. The programmatic method of claim 21 further comprising:generating test card data based on the normalized first truth data and the normalized first detection data; andstoring the test card data in the memory.
23. The programmatic method of claim 22 further comprising providing at least a portion of the test card data to a graphical user interface (GUI).
24. The programmatic method of claim 23 further comprising:receiving user updated test card data from the GUI; andDocket No: 395 / 48 PCTstoring the user updated test card data in the memory.
25. The programmatic method of claim 21 further comprising:updating test card data based on the normalized first truth data and the normalized first detection data; andstoring the program updated test card data in the memory.
26. The programmatic method of claim 25 further comprising providing at least a portion of the program updated test card data to a graphical user interface (GUI).
27. The programmatic method of claim 26 further comprising:receiving user updated test card data from the GUI; andstoring the user updated test card data in the memory.
28. The programmatic method of claim 21 further comprising:generating multi-waypoint test data based on the normalized first truth data and the normalized first detection data; andproviding the multi-waypoint test data to a graphical user interface (GUI).
29. The programmatic method of claim 28 further comprising:receiving updated multi- waypoint test data from the GUI; andstoring the updated multi-waypoint test data in the memory.
30. The programmatic method of claim 1, wherein the first truth data and the first detection data are received in near real time during the first test flight.
31. The programmatic method of claim 1 further comprising providing the first performance data to a graphical user interface (GUI).Docket No: 395 / 48 PCT32. The programmatic method of claim 31, wherein:the first truth data and the first detection data is received in near real time during the first test flight; andthe first performance data is provided to the GUI in near real time during the first test flight.
33. The programmatic method of claim 31, wherein the GUI is provided by a smart phone, a smart tablet, a laptop, a personal computer, or a workstation.
34. The programmatic method of claim 1 further comprising:receiving second truth data associated with a plurality of test flights associated with a plurality UASs;receiving second detection data associated with the plurality of test flights, wherein the second detection data was provided by a plurality of C-UASs monitoring the plurality of test flights;determining second performance data of the plurality of C-UASs based on the second truth data and the second detection data; andstoring the second performance data in a memory.
35. The programmatic method of claim 34 further comprising providing the first performance data and the second performance data to a graphical user interface (GUI).
36. The programmatic method of claim 34 further comprising:normalizing the first truth data and normalizing the first detection data, wherein determining first performance data of the first C-UAS is further based on the normalized first truth data and the normalized first detection data;normalizing the second truth data and normalizing the second detection data, wherein determining second performance data of the plurality of C-UASs is further based on the normalized second truth data and the normalized second detection data; andDocket No: 395 / 48 PCTproviding the first performance data and the second performance data to a graphical user interface (GUI) in a comparative manner between the C-UASs.
37. The programmatic method of claim 36 further comprising:generating test card data based on the normalized first truth data, the normalized first detection data, the normalized second truth data, and the normalized second detection data; and storing the test card data in the memory.
38. The programmatic method of claim 37 further comprising providing at least a portion of the test card data to a graphical user interface (GUI).
39. The programmatic method of claim 38 further comprising:receiving user updated test card data from the GUI; andstoring the user updated test card data in the memory.
40. The programmatic method of claim 37 further comprising:updating test card data based on the normalized first truth data, the normalized first detection data, the normalized second truth data, and the normalized second detection data; and storing the program updated test card data in the memory.
41. The programmatic method of claim 40 further comprising providing at least a portion of the program updated test card data to a graphical user interface (GUI).
42. The programmatic method of claim 41 further comprising:receiving user updated test card data from the GUI; andstoring the user updated test card data in the memory.
43. The programmatic method of claim 36 further comprising:Docket No: 395 / 48 PCTgenerating multi-waypoint test data based on the normalized first truth data, the normalized first detection data, the normalized second truth data, and the normalized second detection data; andproviding the multi-waypoint test data to a graphical user interface (GUI).
44. The programmatic method of claim 43 further comprising:receiving updated multi- waypoint test data from the GUI; andstoring the updated multi-waypoint test data in the memory.
45. The programmatic method of claim 1, wherein the programmatic method is implemented on a smart phone, a smart tablet, a laptop, a personal computer, or a workstation.
46. The programmatic method of claim 45, wherein the programmatic method is implemented as a progressive web application (PWA).
47. The programmatic method of claim 1, wherein the programmatic method is implemented on a server.
48. The programmatic method of claim 47, wherein the server is a portion of a networked computing environment.
49. The programmatic method of claim 47, wherein the programmatic method is implemented on a virtual server.
50. The programmatic method of claim 47, wherein the server comprises an operating system (OS) hosted within a container running on the server and the programmatic method is executed over the OS hosted within the container.
51. The programmatic method of claim 50, wherein the container is a Docker® container.Docket No: 395 / 48 PCT52. The programmatic method of claim 1, wherein the programmatic method is implemented within a spreadsheet application.
53. The programmatic method of claim 1 further comprising storing the first performance data as a spreadsheet compatible file.
54. The programmatic method of claim 53, wherein the spreadsheet compatible file is configured for presenting the first performance data as a table, a chart, an image, or a dashboard.
55. A computing system comprising:a memory; andat least one processor configured, wherein the computing system is configured for:receiving first truth data associated with a first test flight of a first unmanned aircraft system (UAS);receiving first detection data associated with the first test flight, wherein the first detection data was provided by a first counter-unmanned aircraft system (C-UAS) monitoring the first test flight;determining first performance data of the first C-UAS based on the first truth data and the first detection data; andstoring the first performance data in the memory.
56. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium storing instructions to be implemented on at least one computing device including at least one processor, the instructions when executed by the at least one processor cause the at least one computing device to perform a method comprising:receiving first truth data associated with a first test flight of a first unmanned aerial vehicle (UAS);Docket No: 395 / 48 PCTreceiving first detection data associated with the first test flight, wherein the first detection data was provided by a first counter-unmanned aircraft system (C-UAS) monitoring the first test flight;determining first performance data of the first C-UAS based on the first truth data and the first detection data; andstoring the first performance data in a memory.
57. A method for counter-unmanned aircraft system (C-UAS) evaluation, the method comprising:conducting a first test flight of a first unmanned aircraft system (UAS);recording first truth data associated with the first test flight;recording first detection data associated with the first test flight, wherein the first detection data was provided by a first counter-unmanned aircraft system (C-UAS) monitoring the first test flight;providing the first truth data and the first detection data to a computing system, wherein the computing system is configured for determining first performance data of the first C-UAS based on the first truth data and the first detection data; andstoring the first performance data in a memory.
58. The method of claim 57 further comprising:conducting a plurality of test flights with a plurality of UASs;recording second truth data associated with the plurality of test flights;recording second detection data associated with the plurality of test flights, wherein the second detection data is provided by a plurality of C-AUSs monitoring the plurality of test flights;providing the second truth data and the second detection data to the computing system, wherein the computing system is further configured for determining second performance data on the plurality of C-AUSs; andstoring the second performance data in the memory.
