Drone early warning system

Abstract

Systems and methods described herein include a machine learning / artificial intelligence-based system designed to automatically predict threats associated with drone trajectories. A drone early warning system (DEWS) includes an annotation tool, a feature extraction module, a threat classification module, and an alert module. The annotation tool comprises a user interface that may be used by users to associate geolocated assets and / or asset zones with one or more value classifications. Graded asset values are compiled in a training data set to train a machine learning model for identifying trajectories and / or assets. The feature extraction module extracts characteristic features from live trajectory information that is passed to the threat classification module. The threat classification module merges outputs of a model suite to form a threat prediction for one or more identified drone trajectories. The alert module automatically triggers an alert or reaction based on a parameter set.

Description

[0001] Attorney Docket No. 009043.00129\WO NU2024-236-02 Drone Early Warning System

[0002] CROSS REFERENCE TO RELATED APPLICATIONS

[0003]

[0001] This application priority to U. S. Application No. 63 / 754,812 entitled “Drone Early Warning System” filed on February 6, 2025, which is incorporated by reference in its entirety.

[0004] BACKGROUND

[0005]

[0002] Terror groups such as ISIS [1], the PKK [2], Lashkar-e-Taiba (See, https: / / www.indiatoday.in / india / story / drone-attack-initialprobe-lashkar-role-jammu- and-kashmir-police-chief-1820679-2021-06-29), and others are increasingly using drones in various operations. Drones are also becoming a preferred instrument of nation state warfare as evidenced by the war in Ukraine. There is now deep concern that cities will be targeted by drone attacks [3].

[0006]

[0003] However, the skies over a city are regularly traversed by numerous drones. Realtors use drones to get aerial shots of properties for sale [4], insurance companies use drones to look for undeclared pools and property damage [5], sports arenas use drones to capture crowd pictures and game plays [6], and more. A major problem for police and security organizations around the world is to distinguish the few drones that pose a threat from the many that are benign. And this need requires detection of threatening drones to be done as early as possible. As stated by defense experts at the Modern War Institute at West Point. (https: / / mwi.westpoint.edu / understanding-thecounterdrone-fight-insights-from-combat-in-iraq-and-syria / ), “The earlier you detect a threat (drone, rocket, missile, or artillery), the sooner you can alert the force to seek shelter while the air defense operators work to employ their systems to defeat the threat.”

[0007] SUMMARY

[0008]

[0004] The following presents a simplified summary to provide a basic understanding of some aspects of the disclosure. The summary is not an extensive overview of the disclosure. It is neither intended to identify key or critical elements of the disclosure nor to delineate the scope of the disclosure. The following summary merely presents some concepts of the disclosure in a simplified form as a prelude to the description below.Attorney Docket No. 009043.00129\WO NU2024-236-02

[0009]

[0005] Aspects of the disclosure provide effective, efficient, scalable, and convenient technical solutions that address and overcome the technical problems associated with quickly and reliably assessing threats shortly after identifying a drone trajectory.

[0010]

[0006] The problem of early detection and differentiation of threatening drone flight paths from benign drone flight paths is addressed in this disclosure: such as by developing a machine learning model that takes an initial portion (e.g. the first 5, 10, 20, 30 seconds, etc.) of a drone trajectory as input and predicts whether the drone flight path is threatening or not. The smaller the “initial” part, the earlier we can bring a potentially threatening trajectory to the attention of security agencies. But a small initial partial trajectory might be too short to make a good prediction.

[0011]

[0007] Over the last few years, drone use by terror groups has been increasing, including in armed conflict. Several technologies have been developed to detect drone flights. However, much less work has been done on the Drone Threat Prediction Problem (DTPP): e.g., predicting which drone trajectories are threatening and which ones are not. A Drone Early Warning System (DEWS), may be used as a framework to solve this problem. Solving the DTPP early (e.g., soon after trajectory detection) is key. Once a drone starts on its trajectory, DEWS can make accurate predictions within 20- 30 seconds of the flight with an F1-score of over 80% on data about a major European city. Additionally, the tradeoff between earliness of predictions and accuracy are discussed below. Key features that ensure good predictions are identified, as well.

[0012]

[0008] Though there has been a great deal of work on predicting trajectories of moving objects (e.g. mobile phones [7], drones [8]), relatively little work has been performed on quantifying the threat posed to a city or geographic area by a drone. To quantify this threat, multiple factors in addition to the drone’s trajectory must be analyzed including, for example, the drone’s capabilities (e.g. payload, battery life, max speed) and the value of the assets on the ground that the drone is flying over.

[0013]

[0009] The Drone Early Warning System predicts whether a drone trajectory is threatening or not. DEWS tries to understand a length of observation time of a drone flight is needed to predict whether the drone poses a threat or not.

[0014]

[0010] DEWS is novel in several respects, (i) As far as is known, DEWS is the first framework to predict the threat a drone flight poses to a city, (ii) It is the first framework to understand the tradeoff between the time for which a drone trajectory is observed (e.g.,Attorney Docket No. 009043.00129\WO NU2024-236-02 the “observation window”) and threat prediction accuracy, (iii) In addition to the trajectory, DEWS looks at features about the drone’s capabilities, violations of no-fly zones, assets on the ground, and more, (iv) DEWS identifies the key features linked to accurate predictions. The values of assets on the ground that a trajectory flies over constitute the single most important feature in assessing the threat of the trajectory, (v) DEWS can make predictions with an F1 score exceeding 0.8 in 3 seconds in operational use (after training), suggesting that DEWS may be used for real-time predictions, (vi) DEWS has been tested by Dutch police, municipal, and security officials on 8 months of real trajectories over The Hague and the results show an F1-score over 0.85.

[0015]

[0011] An illustrative method for evaluating and predicting a possibility of threats associated with drone flights track in real-time includes extracting, in real-time based on a drone trajectory module, drone trajectory features characterizing a drone trajectory. The method further includes classifying, by a threat classification model based on the drone trajectory features, the drone trajectory with a predicted threat classification. The method further includes triggering, when the predicted threat classification meets a threshold, a threat assessment alert. Real-time extraction of drone trajectory features enables rapid assessment of potential threats posed by drones flying over protected areas, allowing security personnel to respond promptly to dangerous situations before they escalate. The method may further include training, based on drone trajectory information captured from historical drone flights, the drone trajectory module. Training the drone trajectory module using historical drone flight data improves the accuracy of threat predictions by enabling the system to learn patterns associated with threatening and non-threatening drone behaviors from real-world examples. The drone trajectory information may comprise annotated information corresponding to classified geolocated features of a municipality.

[0016]

[0012] Utilizing annotated information corresponding to classified geolocated features of a municipality allows the system to incorporate expert knowledge about asset values and sensitive locations, which enhances the relevance and accuracy of threat assessments for specific geographic areas. The drone trajectory features may comprise drone positioning features and geolocated asset features. Incorporating both drone positioning features and geolocated asset features enables the system to evaluate threats based not only on the drone's flight path but also on the value of assets on the ground that the drone flies over, which has been identified as a primary determinant of threatAttorney Docket No. 009043.00129\WO NU2024-236-02 level. The classifying may be performed based on a suite of machine learning classifiers, where employing the suite of machine learning classifiers provides diverse analytical perspectives on the trajectory data. This increases the robustness of threat predictions by leveraging the strengths of multiple classification approaches.

[0013] The classifying may include combining a threat prediction from a plurality of machine learning classifiers. By combining these threat predictions through fusion techniques, the system improves overall classification accuracy by integrating complementary insights from different models, reducing the likelihood of missed threats or false alarms. In some cases, the combining of the threat predictions may further include a weighted sum of probability estimates of a plurality of predictions from the plurality of machine learning classifiers. Using a weighted sum of probability estimates allows the system to assign greater influence to the classifiers that demonstrate superior performance, thereby optimizing the final threat prediction based on the relative reliability of each model.

[0017]

[0014] In some cases, the system and method may provide forecasted threat assessments based on limited real-time data (e.g., about 5 seconds, about 10 seconds, etc.) of an observed drone trajectory. Continually providing forecasted threat assessments based on short portions of observed drone trajectory enables early warning capabilities that allow security officials to initiate response protocols while a drone flight is still in its initial stages. In this way, the threat assessments may be tailored to a specific context of a drone flight, which may allow the systems and methods to automatically adapt its predictions based on situational factors, improving the relevance of alerts for different operational scenarios, and / or geographic locations.

[0018]

[0015] The " Related Work" section discusses related work. The next section, " DTPP: Drone Threat Prediction Problem" formalizes the problem studied. The " DEWS Architecture" section provides a detailed description of the architecture, including its features and the training process. The " Experiments" section presents the predictive performance of 11 ML models and a late fusion classifier) as the observation (e.g., a training cycle) window increases. After this, a " Limitations and Future Work" section describes limitations of the framework.Attorney Docket No. 009043.00129\WO NU2024-236-02

[0019]

[0016] The details of these and other aspects of the disclosure are set forth in the accompanying drawings and description below. Other features and advantages of the disclosure will be apparent from the drawings and description.

[0020] BRIEF DESCRIPTION OF THE DRAWINGS

[0021]

[0017] The foregoing and other objects, features, and advantages of the present disclosure set forth herein will be apparent from the following description of particular embodiments of those inventive concepts, as illustrated in the accompanying drawings. Also, in the drawings the like reference characters refer to the same parts throughout the different views. The drawings depict only typical embodiments of the present disclosure and, therefore, are not to be considered limiting in scope.

[0022]

[0018] FIG. 1 shows an illustrative sample drone trajectory with a thirty-second restriction, according to aspects of the present disclosure;

[0023]

[0019] FIG. 2 shows an illustrative Drone Early Warning System (DEWS), according to aspects of the present disclosure;

[0024]

[0020] FIG. 3 shows a table comprising illustrative DEWS dataset statistics, according to aspects of the present disclosure;

[0025]

[0021] FIGS. 4A-C show illustrative High-Threat Prediction (HTP) settings, according to aspects of the present disclosure;

[0026]

[0022] FIGS. 5A, 5B, and 5C show charts corresponding to an illustrative ablation study, according to aspects of the present disclosure;

[0027]

[0023] FIG. 6 shows a feature relevance analysis of an illustrative HTP problem, according to aspects of the present disclosure;

[0028]

[0024] FIG. 7 shows charts associated with an illustrative HTP problem, according to aspects of the present disclosure;

[0029]

[0025] FIG. 8 shows an illustrative block diagram of a processor platform implementing at least a portion of the drone early warning system, according to aspects of the present disclosure;Attorney Docket No. 009043.00129\WO NU2024-236-02

[0030]

[0026] FIGS. 9A-9C show the performance of DEWS under the Low -Threat Prediction (LTP) setting, according to aspects of the present disclosure;

[0031]

[0027] FIGS. 10 A- 10C show the performance of DEWS under the Medium-Threat Prediction (MTP) setting, according to aspects of the present disclosure;

[0032]

[0028] FIGS. 11 A and 11B show a table of DEWS features categories and descriptions, according to aspects of the present disclosure; and

[0033]

[0029] FIG. 12 shows a table of additional DEWS features and categories and descriptions, according to aspects of the present disclosure.

[0034] DETAILED DESCRIPTION

[0035]

[0030] Aspects of the present disclosure relate to computing methodologies such as computer vision, natural language processing, artificial intelligence, and applied computing. More particularly the present disclosure relates to a framework to generate engaging multimodal image-text pairs, with a focus on maximizing user engagement while maintaining the relevance of the content to the given topic.

[0036] Related Work

[0037]

[0031] Predicting the future location of a moving object has been explored in various domains [9],

[0010] . Vision- based object tracking methods

[0011] predict the future location of moving objects. This work has been used in self-driving cars

[0010] to create plans based on predicted future locations of humans and nearby moving objects. Other research uses historical GPS data to predict mobility of devices [7].

[0038]

[0032] Numerous papers predict vehicle trajectories by learning models from historical driving data

[0012] , Temporal models such as LSTMs with attention networks

[0013] ,

[0014] ,

[0015] have been proposed for trajectory pre- diction. Recent advances incorporate trajectories of nearby vehicles to reduce accidents

[0016] . Drone trajectory prediction has been widely studied across various applications, including autonomous aerial cinematography

[0017] , delivery [8], and search and rescue

[0018] ,

[0039]

[0033] There is also work on predicting a mobile phone’s next location based on historical movement data

[0019] ,

[0020] , These approaches include sequential pattern learning techniques to predict a phone’s future location and / or human movements.Attorney Docket No. 009043.00129\WO NU2024-236-02

[0034] DEWS differs from past efforts in two respects. First, it predicts if a drone trajectory is threatening or not, which past works do not do. Second, DEWS is the first to study how early in a trajectory a good prediction can be made. This is particularly important because timeliness is key in mitigating drone threats. The identification of a threat is crucial input for the subsequent command and control process resulting in some kind of intervention. DEWS not only obtains features from the drone trajectory, but also from information about assets on the ground and the drone’s capabilities. Past work does not consider assets on the ground.

[0040] DTPP: Drone Threat Prediction Problem

[0041]

[0035] FIG. 1 shows an illustrative drone trajectory map 100 with its 30-second restriction.

[0042] The trajectory data is from a real drone, but the city was altered for security reasons. Suppose C is a city to be protected. A map of C containing locations of important national buildings, security installations (e.g., police stations, military bases), government buildings, hospitals, tourist attractions, entertainment venues, homes, parks, roads, bridges, utilities, etc., is obtained such as, for example, from OpenStreetMap at https: / / www.openstreetmap.org. Once the city C is selected, an asset valuation map Val(C) is defined, which assigns a value to every point within the city. High Val(C) values corresponds to important locations.

[0043]

[0036] Consider a drone d flying over C. Its trajectory rdis a finite sequence (-3^, G),...,

[0044]

[0045] t„) where each 3^ = (Zat;, longt, altt) is d's location at time t, in terms of latitude, longitude, and altitude, respectively. The temporal restriction of a trajectory Tdto time j, denoted tr(Td,j ), is the set

[0046]

[0047] {(3f, ) | tt) G t / \ tt< j}. T denotes a given set of trajectories and tr (7', j) = {Ir (T, j) 1 1 G T } to be the restriction of the trajectories in T to the first j timepoints. The drone trajectory map of FIG. I shows a drone’s trajectory Tdand its restriction tr(d, 30 ) to 30 seconds. Drone locations may be acquired at irregular intervals. As an example, a level of threat posed by tr(tr(τd, 30) after the 30 seconds of the flight may be predicted. The threat score is given by y(τd,) G [1, 10]. The higher the threat score, the more threatening the drone’s trajectory.

[0048]

[0037] The Drone Threat Prediction Problem (DTPP[lev]) is to learn a function flev: (d, tr (τd, j)) → {0, 1 }, such that f(d, tr (τd, j)= 1 if the threat posed by τj≥ lev, where lev G [1, 10],Attorney Docket No. 009043.00129\WO NU2024-236-02

[0049]

[0038] DTPP can work after any observation window j > 0 after the drone flight begins. This is critical for security. The earlier predictions are made about the threat level of trajectories, the earlier security officials can prioritize their responses, as indicated by the Modern War Institute at West Point. Earliness of prediction must be balanced against accuracy of prediction. Understanding this balance is a major goal of this paper. DEWS Architecture

[0050]

[0039] FIG. 2 shows an illustrative DEWS architecture (e.g., a DEWS enabled computing system 200), according to aspects described herein DEWS uses a dataset of drone trajectories annotated by Dutch police and municipality, of which Table 1, as shown in FIG. 3, presents a brief overview. The DEWS enabled computing system 200 may include an annotation tool 210 having one or more inputs 220, a training data store 230, a feature extraction module 240, a threat classification module 250, and a response alert module 270.

[0051]

[0040] In some cases, the one or more inputs 220 may allow one or more computing devices of security experts, or other individuals, to annotate a map such as to classify various geographic and / or building details. With respect to FIG. 2, data set preparation involves annotating a map showing aerial views of the geographic features, via the annotation tool 210, with asset values and drone trajectories by experts (e.g., police personnel, security personnel, etc.). For example, assets 212 (or areas within a specified distance of an asset), no-fly zones 214, and / or other features may be annotated as shapes (e.g., rectangles), as shown on a screen of the annotation tool 210. Subsequently, DEWS extracts features and trains eleven classifiers M1, · · · , M11 to yield eleven predictions which are integrated using late fusion to predict the final threat level. During operational use (e.g., after training), an initial part of a live trajectory is processed to extract features, and the combination of single predictors and late fusion produces the final threat score.

[0052]

[0041] The feature extraction module 240 extracts key features that characterize a drone trajectory. As shown, the feature extraction module 240 may use information sourced from the training data store 230 to analyze a live trajectory 245 fed into an input of the feature extraction module 240 in real-time. The threat classification module 250 combines the predictions of eleven classifiers to provide a final classification. ForAttorney Docket No. 009043.00129\WO NU2024-236-02 example, the threat classification module may utilize one or more feature subsets 252 as inputs into each model of a model suite whose outputs may be fused by a fusion module 246 to generate predicted threat associated with the live trajectory 245.

[0053] Trajectory Training Data

[0054]

[0042] A dataset of three hundred forty -nine drone trajectories was collected to train DEWS.

[0055] In this example, the collected trajectories represent all known recorded drone flights over The Hague captured by Dutch police and by the municipality over a period of eight months. The threat associated with each trajectory was assessed on a 1-10 scale by at least one police official. Fifty trajectories were annotated independently by two or more police and municipal officers. To assess agreement amongst the officials, the interannotation weighted Cohen’s kappa coefficient of 0.772 was calculated, indicating substantial agreement amongst annotators.

[0056]

[0043] Police officials then categorized trajectories as low threat (score < 4), medium threat (score G [4, 8)), and high threat (score > 8).

[0057] Feature Extraction

[0058]

[0044] This feature extraction module 240 extracts one hundred ten features for each trajectory.

[0059] Basic features offer an initial summary of each trajectory. They include the number of observations, duration of the flight, distance traveled, and communication channel used (e.g. radio-frequency, Wi-Fi, etc.).

[0060]

[0045] Capability features of a drone may include physical attributes (e.g., weight, dimensions) and / or performance specifications (e.g., maximum payload, battery capacity). These features are used for assessing the drone’s operational limits and the potential threat it may pose. DEWS processes data on drone trajectories, which includes information about the drone type. This information may be cross-referenced with information stored in a data store (e.g., an internal database) that maps each drone type to its specific characteristics. Such characteristics may include model configuration information, payload capacity information, range information, and the like. In some cases, an image analysis tool that may be incorporated within the feature extraction module 240 may be used to identify and / or confirm characteristics of a subject drone. In some cases, the image analysis tool may capture and / or supplement data before providing input data toAttorney Docket No. 009043.00129\WO NU2024-236-02 DEWS. Further, DEWS may utilize aspects of third-party systems, such as by being built on top of a third-party system that can identify the trajectory of a drone. Such a system may be based on many possible drone identification techniques, such as by identifying communication frequencies being used between a drone and its controller, identifying visual features of a drone provided by an image analysis system, analyzing and / or identifying audio signatures from an audio signal captured by an audio analysis system, and / or the like. In some cases, DEWS may utilize some combination third- party systems to provide and / or supplement input data.

[0061]

[0046] Altitude features (e.g., the mean altitude above takeoff) and speed features (e.g., minimum / maximum speeds) provide insight into the dynamics of each trajectory. They are essential for detecting suspicious activities and in ensuring regulatory compliance as drones may have altitude or speed restrictions.

[0062]

[0047] No-fly Zone features capture the behavior of trajectories in terms of their respect for the law. The website https: / / www.godrone.nl was used identify no-fly zone data and six features to quantify the proximity of the trajectory to a no-fly zone were defined. For example, this information was used to identify whether the drone entered a no-fly zone, the percentage of time the trajectory was within a no-fly zone, and the like.

[0063]

[0048] Asset-based features (e.g. dams, utilities, government buildings, defense sites) also need to be considered when assessing the threat of a trajectory. Features were defined about the proximity of the trajectory to these assets (e.g. the maximum / mean asset values overflown). In some cases, asset values may be provided by experts, for example, as annotations (e.g., security experts such as Dutch police). When annotations are missing, asset information and / or feature information may be retrieved via one or more tools and / or applications such as OpenStreetMap. These tools may provide additional features and / or information that may be utilized by DEWS. Additionally, DEWS may utilize A I / ML models to predict an asset value based on human annotations as a ground truth. In some cases, a set or partial set of human annotations associated with one or more cities may be used to infer annotations of assets for other cities (e.g., cities that do not have human annotations and / or may have partial annotations) using AI / ML models.Attorney Docket No. 009043.00129\WO NU2024-236-02

[0064]

[0049] Observation history features capture the similarity between the current trajectory and historical trajectories. Self-similarity features refer to the similarity between the current trajectory and past trajectories of the same drone, which can help detect recurring flight patterns or behaviors that may indicate potentially benign operations. Cross-similarity features capture the similarity between the current trajectory and past trajectories of other drones. This may be useful for identifying anomalous behavior by comparing it to known suspicious or dangerous flight patterns exhibited by other devices. Cosine similarity is used in both.

[0065] Threat Classification

[0066]

[0050] The threat classification module 250 predicts the threat level (e.g., low, medium, high) of a trajectory based on its extracted features. To accomplish this, a suite of eleven well-known machine learning classifiers (e.g., models of the model suite 254) was trained, encompassing both traditional and neural network models. The classifiers used in DEWS are: Logistic Regression, k-Nearest Neighbors (KNN), Support Vector Machines (SVM), Decision Trees, Random Forest, Gradient Boosting, Naive Bayes, AdaBoost, Extra Trees, a Multi-layer Perceptron (MLP), and a wide-and-deep neural network.

[0067]

[0051] For each classifier, hyper-parameter optimization and applied feature selection were used to identify the most relevant subset of features (e.g., the feature subsets 252). The feature selection process consists of three main steps: (1) removing constant columns, (2) retaining only one feature from pairs with a Pearson correlation greater than 0.95, and (3) selecting the top- / < features based on their Mutual Information (MI) scores, where k is a user defined parameter. This approach allowed us to develop specialized models that leverage distinct subsets of features for the same trajectory, thereby enhancing model diversity within the suite.

[0068]

[0052] After individually training each model Mi, late fusion was used to combine their predictions. The final threat score for a trajectory t is computed as a weighted sum of the probability estimates produced by each model: y(t) =

[0069]

[0070] ■ w;, where Mfit) represents the probability prediction of model Mi for trajectory t, and wLdenotes the weight assigned to model Mi. The weights lywere optimized through grid search to identify the combination of weights that maximized overall classificationAttorney Docket No. 009043.00129\WO NU2024-236-02 performance. This fusion process enables DEWS to integrate the strengths of multiple models of the model suite 254, ensuring robust and accurate threat classification. Experiments

[0071]

[0053] All experiments were conducted on a computational platform having a 9th Gen Intel i9-10980XE processor, 256 GB of RAM, and an NVIDIA RTX A6000. The codebase involved approximately 2000 lines of code in Python 3.10. All classification models were implemented using the Scikit-learn library, expect for the wide and deep classifier for which we used the Tensorflow 2 library. Other computing devices, programming languages, artificial intelligence / machine learning libraries, and / or classifiers may be used without departing from the methods and / or procedures discussed within this disclosure and the examples discussed are not limiting.

[0072] Data Collection

[0073]

[0054] Data about 349 drone trajectories over a Dutch city was systematically collected by the Dutch police using a drone tracking system (e.g., the Senhive commercial drone tracking system, https: / / senhive.com / sen-id-l). Such drone tracking systems track drones by monitoring their communication frequencies with drone operators, allowing for the detection and recording of their trajectories within a defined radius (e.g., 25 km). An anonymized version of this dataset was provided to the academic part of our team, with sensitive information such as device IDs replaced with anonymized IDs. Summary statistics for the dataset are provided in Appendix below.

[0074]

[0055] The DEWS system includes GUIs for annotating asset values and threat scores associated with the drone trajectories. When this paper is published, we will release anonymized versions of the DEWS data.

[0075] Experimental Protocol

[0076]

[0056] In the experiments, the DTPP problem was addressed at three distinct levels (e.g., one or more thresholds 262): 3, 5 and 7. This corresponds to the different threat prediction scenarios detailed as follows:

[0077]

[0057] (i) “Low-Threat” Prediction (LTP): trajectories with a threat score in the [3, 10) range (i.e. greater than or equal to 3 and strictly less than 10) are considered low-threats. TheAttorney Docket No. 009043.00129\WO NU2024-236-02 LTP problems predicts “no-threat” at a score less than 3 and low-threat at a score greater than 3.

[0078]

[0058] (ii) “Medium-Threat” Prediction (MTP): trajectories with a threat score in the [5, 10) are considered medium-threat trajectories. So MTP distinguishes between mediumthreats, at a score of 5 or more, and other trajectories.

[0079]

[0059] (iii) “High-Threat” Prediction (HTP): trajectories with a threat score greater than or equal to 8 are classified as threatening (e.g., high-threat), while trajectories with a score of 7 or less are classified as non-threatening.

[0080]

[0060] By applying the learned predictive models for a given trajectory, trajectories can be uniquely classified into one of the four threat levels (no threat, low threat, medium threat, and high threat). The response module 270 may perform one or more actions based on a particular threat prediction value input based on the thresholds 262.

[0081]

[0061] These classification tasks are increasingly difficult due to the skewed distribution of threat labels, where the HTP setting contains significantly fewer threatening trajectories compared to MTP and LTP.

[0082]

[0062] Three illustrative experiments were conducted:

[0083] a. Early Threat Prediction Evaluation: The DEWS’s capability is assessed for early threat prediction by varying the observation window for each trajectory. Specifically, each trajectory t was analyzed using the first i seconds of a flight, where i G {1, 5, 10, 20, 30, 60, 180, 360, 720}. This assesses how early accurate predictions about the potential threat can be made.

[0084] b. Ablation Study: To determine the relative importance of different feature types, each feature type was systematically removed from the model and retraining was performed on the DEWS [If] late fusion predictor. Performance is then evaluated based on recall, precision, and Fl -scores to identify which features contribute most significantly to predictive accuracy.

[0085] c. Feature Relevance Analysis: Assuming that features selected for classification are the most relevant for solving the task, we analyze the features chosen by each classifier during the feature selection process. For each observationAttorney Docket No. 009043.00129\WO NU2024-236-02 window, we count how often each attribute is selected for classification across all classifiers in the model suite. These counts are then normalized to compute the relative frequency of each feature category. Specifically, let w denote an observation window, A = Fi, F,..., Fn} represent the set of features and Mi, M2,..., Mu be the classifiers in the model suite. Further definitions include:

[0086]

[0063] - as the count of how often the feature Ft is selected for classifier Mj during Mj ’s features optimization process for classification, within window vv;

[0087]

[0064] - N.w>as the total count of how often the feature F, is selected across all classifiers, e.g.,

[0088]

[0089] ^(w)= S“o N^\

[0090]

[0065] To compute the relative frequencyw)f (w) i of the feature i for the observation window w, we normalize N.w>by the total counts for all attributes:

[0091] f (w) _ _jvp__

[0092]

[0093]

[0066] • Runtime: The DEWS’s runtime for feature extraction and classification with late fusion in operational use (after training) is measured. All experiments were conducted using time series cross-validation, a model was learned from an early set of trajectories and then used them to predict on later sets of trajectories.

[0094] Results

[0095] Early Threat Prediction Evaluation

[0096]

[0067] FIGS. 4A-C illustrate DEWS ’ s performance under the HTP setting. Specifically, FIGS.

[0097] 4A, 4B, and 4C depict precision, recall, and Fl -score, respectively, for all eleven classifiers as well as the DEWS late fusion classifier, DEWS[lf]. These metrics are analyzed by varying the observation window. Performance comparisons under MTP and LTP settings are reported in the Appendix.

[0098] Finding 1: Late Fusion is the Best Predictor.

[0099]

[0068] Late fusion consistently outperforms the eleven classifiers across all observation windows, achieving the highest results in terms of precision, recall, and Fl -score. As shown in FIGS. 4A-C, within an observation window of 1-30 seconds, DEWS [If] stabilizes at an Fl -score of approximately 80%, with precision exceeding 90% andAttorney Docket No. 009043.00129\WO NU2024-236-02 recall around 75%. As the observation window increases, performance shows an upward trend, with the most substantial improvement occurring between one minute and three minutes. The best performance is achieved at the six-minute threshold, where precision reaches 0.967 and recall 0.869.

[0100] Finding 2: Increasing the Observation Window may Not Improve Performance.

[0101]

[0069] Interestingly, increasing the observation window does not always lead to improved performance. For example, FIGS. 4A-C show that the highest recall of 0.789 for shorter observation windows occurs with 5 seconds of observation, when precision is 0.934 (using our DEWS[lf] classifier). Both metrics show a slight decline when the window is extended up to 30 seconds. Moreover, beyond six minutes, performance deteriorates across all models and metrics.

[0102] Finding 3: Precision is always higher than recall.

[0103]

[0070] FIGS. 4A-C show that the same time thresholds yield higher performance in terms of precision compared to recall. High-Threat Prediction (HTP) settings: Precision (chart 400), Recall (chart 420), and Fl -score (chart 440) metrics are shown as functions of varying temporal restrictions on the trajectories. The top row provides a zoomed-in view of the results for shorter time windows (less than 30 seconds), while the bottom row displays the complete range of observation windows. For instance, with a short observation window of 5 seconds, precision reaches 0.934, while recall is comparatively lower at 0.789. This trend is consistently observed across all observation windows. This is due to the imbalance of the data considered for the HTP problem which causes DEWS to be more conservative when predicting the highly threatening (minority) class. This suggests that DEWS is very accurate at detecting highly threatening trajectories with a very low false positive rate.

[0104]

[0071] This is extremely valuable for police for two critical reasons: First, it enhances trust in the system, as the low false positive rate minimizes the likelihood of unnecessary interventions. Second, in resource constrained environments, human assessment of predicted high threat can be costly and inefficient. High precision ensures that humans do not get frustrated with false positives.

[0105] Ablation StudyAttorney Docket No. 009043.00129\WO NU2024-236-02

[0106]

[0072] FIGS. 5A, 5B, and 5C show the Fl-scores obtained when removing individual feature categories under the LTP, MTP, and HTP settings, respectively. For example, the ablation study shows Fl -score under LTP (chart 500 of FIG. 5 A), MTP (chart 520 of FIG. 5B) and HTP (chart 540 of FIG. 5C) settings when removing one feature category. The dashed line represents the scenario with all features.

[0107]

[0073] Finding 4: Asset-related features are the most critical for threat prediction. In FIGS.

[0108] 5A-5C, with a 5-second observation window, the Fl -score with all features included is 0.723 in the HTP setting, but decreases to 0.586 when asset features are excluded, representing a 19% reduction in performance.

[0109]

[0074] Equally surprising are the features that proved to be less important than had been expected. For example, an initial hypothesis was that no-fly zone features would play a significant role in threat prediction, yet they had a relatively minor impact on the model’s performance. Similarly, the type of drone (e.g., fast drones with large payloads) was expected to be a key predictor, but their importance for prediction was small. Additionally, speed-related features, which were assumed to be important, turned out to have limited significance in the experiments.

[0110]

[0075] Overall, these findings support the preliminary hypothesis that the geographical region, represented by asset-related features, is a key determinant in assessing the threat level of a trajectory, independent of the drone’s intrinsic characteristics or the specific properties of the trajectory itself.

[0111] Feature Relevance Analysis

[0112]

[0076] The results shown in the chart 600 of FIG. 6 (HTP problem) indicate that as the observation window increases, the importance of asset-related features becomes more pronounced. For instance, after an 180 second observation window, over 60% of the features used for classification belong to the asset category. Interestingly, within the first 1-5 seconds of observation, capability-related features show relatively high importance. The importance of these features decreases sharply with longer observation windows. This may be due to the limited information available in short observation windows, where the drone’s capabilities alone serve as a strong indicator of potential threat.Attorney Docket No. 009043.00129\WO NU2024-236-02 Runtime

[0113]

[0077] FIG.7 shows the mean DEWS runtime for feature extraction (e.g., chart 700) and prediction (with late fusion, e.g., the chart 750) under the HTP setting in operational use. The feature extraction time shows a slight increase with a larger observation window, reflecting the additional computational load due to the increased number of trajectory points. In contrast, the prediction time is not affected by the length of the trajectory. With an overall classification time of approximately 3 seconds, the DEWS system demonstrates its potential for real-time predictions, enabling trajectory classification after just 3-5 seconds of observation. This highlights the system’s suitability for applications requiring prompt decision-making.

[0114]

[0078] FIG. 8 is a block diagram of an illustrative processor platform 800 structured to executing the instructions of at least FIG. 2 to implement the illustrative components, operations, and / or actions to generate information as disclosed and described herein with respect to FIGS. 1-7. The processor platform 800 can be, for example, a server, a personal computer, a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPadTM), a personal digital assistant (PDA), an Internet appliance, and / or any other type of computing device or combination of computing devices.

[0115]

[0079] The processor platform 800 includes a processor 812. The processor 812 is hardware.

[0116] For example, the processor 812 can be implemented by integrated circuits, logic circuits, microprocessors or controllers from any desired family or manufacturer.

[0117]

[0080] The processor 812 includes a local memory 813 (e.g., a cache). The processor 812 of FIG. 8 executes the instructions associated with functionality of at least FIG. 2 to implement the systems and infrastructure and associated methods of FIGS. 1-7 such as one or more computing devices of the illustrative DEWS enabled computing system 200, etc. The processor 812 is in communication with a main memory including a volatile memory 814 and a non-volatile memory 816 via a bus 818. The volatile memory 814 can be implemented by Synchronous Dynamic Random- Access Memory (SDRAM), Dynamic Random-Access Memory (DRAM), RAMBUS Dynamic Random-Access Memory (RDRAM) and / or any other type of random-access memory device. The non-volatile memory 816 can be implemented by flash memory and / or any other desired type of memory device. Access to the one or more main memoryAttorney Docket No. 009043.00129\WO NU2024-236-02 components(e.g., the volatile memory 814, the non-volatile memory 816) may be controlled by a clock controller.

[0118]

[0081] The processor platform 800 of the illustrated example also includes an interface circuit 820. The interface circuit 820 can be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and / or a PCI express interface.

[0119]

[0082] In the illustrated example, one or more input devices 822 are connected to the interface circuit 820. The one or more input devices 822 permits a user to enter data and commands into the processor 812. The one or more input devices 822 can be implemented by, for example, a sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint, and / or a voice recognition system.

[0120]

[0083] One or more output devices 824 are also connected to the interface circuit 8208. The output devices 824 can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touchscreen, a tactile output device, and / or speakers). The interface circuit 8208, thus, typically includes a graphics driver card, a graphics driver chip, or a graphics driver processor.

[0121]

[0084] The interface circuit 820 also includes a communication device such as a transmitter, a receiver, a transceiver, a modem and / or network interface card to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 826 (e.g., an Ethernet connection, a digital subscriber line (DSL), a telephone line, coaxial cable, a cellular telephone system, etc.).

[0122]

[0085] The processor platform 800 also includes one or more mass storage devices 828 for storing software and / or data. Examples of such mass storage devices 828 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, RAID systems, and digital versatile disk (DVD) drives, and / or other mass storage devices.

[0123]

[0086] The coded instructions 832 of FIG. 8 can be stored in the mass storage device 828, in the volatile memory 814, in the non-volatile memory 816, and / or on a removable tangible computer readable storage medium such as a CD or DVD.Attorney Docket No. 009043.00129\WO NU2024-236-02 Additional features

[0124]

[0087] Like all studies, the study can be improved in many ways. First, the study looked at all trajectories over a city that Dutch police tracked over an 8-month period. But these may not reflect all possible flights, for example, because of limitations in tracking technology. Second, once adversaries know about DEWS, they may take evasive actions to prevent their intentions being predicted. To some extent, this is mitigated by the finding that asset value is far and away, the important feature in assessing threat - and adversaries cannot manipulate that. In some cases, development of ML models that are more robust to an adversary’s evasion attempts may be utilized to identify one or more drone trajectory and / or capability obfuscation as part of an adversary’s evasion attempt. Third, there is the possibility of collusion. For example, two or more trajectories that individually seem non-threatening might collude to pose a significantly higher threat. In such cases, an AI / ML model may be used to identify potential threats when multiple trajectories are identified.

[0125] Conclusion

[0126]

[0088] To the best of our knowledge, this is the first paper to explicitly study the problem of how threatening a drone flight is to a city or geographic region. We propose a repertoire of features for quantifying the threat of a drone flight, build out the first drone threat dataset that was assessed by police and security officials and will be made publicly available (with some anonymization to ensure security), and build the first predictive models to assess the threat level posed by a trajectory. We are also the first to show that we can predict threat levels early, when a trajectory is just underway. With just 5- 10 seconds of trajectory data, DEWS is able to make predictions of high threat levels with an Fl -score over 0.8. And these predictions take only a few seconds to make. Predictive accuracy goes up till about 5-6 minutes of the trajectory is observed. This enables DEWS to continuously provide forecasts to security officials after 5 seconds of the flight is observed and they can decide on their response depending on their own judgement and knowledge of context. DEWS also allows predictions to be tailored to a specific context and threat assessment. In other words, given a specific threat assessment, particular assets (on the ground) or capabilities (of drones) may be valued differently — and DEWS will still work.Attorney Docket No. 009043.00129\WO NU2024-236-02

[0127]

[0089] A major new finding is that the key determinant of the danger posed by a trajectory is not the trajectory itself, but the values of the assets on the ground that a trajectory flies over.

[0128]

[0090] From the foregoing, it will be appreciated that the above disclosed methods, apparatus, and articles of manufacture have been disclosed to improve the functioning of a computer and / or computing device and its ability to identify a threat level associated with a drone flight trajectory over a city.

[0129]

[0091] Certain examples provide a machine leaming / artificial intelligence-based system designed to automatically predict threats associated with drone trajectories. A drone early warning system (DEWS) includes an annotation tool, a feature extraction module, a threat classification module, and an alert module. The annotation tool comprises a user interface that may be used by users to associate geolocated assets and / or asset zones with one or more value classifications. Graded asset values are compiled in a training data set to train a machine learning model for identifying trajectories and / or assets. The feature extraction module extracts characteristic features from live trajectory information that is passed to the threat classification module. The threat classification module merges outputs of a model suite to form a threat prediction for one or more identified drone trajectories. The alert module automatically triggers an alert or reaction based on a parameter set.

[0130] Appendix

[0131] DEWS Features

[0132]

[0092] For each trajectory t, DEWS captures the following types of features. It is important to note that the data used by DEWS was captured in real-time by the Dutch police using third party tools to monitor drone communications. Thus, all of the features listed below are available at or before the time of a flight based on available features captured by an illustrative toolset. Details (including units) are shown in FIGS 11 A, 11B, and 12. FIGS. 11 A and 1 IB show a table, split between the two pages, of DEWS features, categories, and descriptions. FIG. 12 shows a table of additional DEWS features and categories and descriptions.Attorney Docket No. 009043.00129\WO NU2024-236-02

[0133]

[0093] Basic Features. These include the number of observations, the duration of the flight, the distance traveled, and the communication channel used (e.g. radio frequency, WIFI, etc.).

[0134]

[0094] Drone Capability Features. These include the weight, dimensions, max payload, max ascent speed, max descent speed, max horizontal speed, max takeoff altitude, max flight time, max hovering time, max flight distance, max windspeed resistance, max pitch angle, and battery (e.g., type, weight, capacity, etc.), and the like.

[0135]

[0095] Asset Features. Asset features assign values to different parts of a city being protected.

[0136] For the illustrative example, these features were assigned to regions of the city by Dutch police officers in advance. The features listed here can be extracted using the prespecified city values and the given drone trajectory. These include, among other features, the maximum value of assets on the ground that the drone has flown over as well as features depending on the distributions of asset values around each point in the trajectory.

[0137]

[0096] Altitude Features. These include the altitude at both the start and end of the drone’s trajectory, the mean altitude throughout the trajectory, the standard deviation of the altitude, and additional metrics based on the distribution of altitude values along the trajectory.

[0138]

[0097] No-fly Zone Features. No-fly Zone features are based on areas within the city where drone flight is restricted. These features include, among others, indicators of whether the drone entered a no-fly zone, the percentage of trajectory points within such zones, and features representing the distance from the drone to the nearest no-fly zone.

[0139]

[0098] Speed Features. These include the speed of the drone at both the start and end of its trajectory, the mean speed throughout the trajectory, the standard deviation of the speed, and additional metrics based on the distribution of speed values along the trajectory. In some cases, drone speed at or near assets may be identified.

[0140]

[0099] Observation History Features. These include the distance between the current trajectory and the closest (and past) trajectories from the same drone (self-similarity) or other observed drones (cross-similarity), as well as the threat scores associated with these trajectories.Attorney Docket No. 009043.00129\WO NU2024-236-02 Early Threat Prediction Evaluation

[0141]

[0100] FIGS. 9A-9C illustrate the performance of DEWS under the Low-Threat Prediction (LTP) setting, while FIGS. 10A-10C present the results for the Medium-Threat Prediction (MTP) setting. For both threat levels, the evaluation metrics — precision, recall, and F1-score — are assessed across different observation windows. FIGS 9A-9C show low-threat prediction (LTP) settings, such as Precision in FIG. 9A, Recall in FIG. 9B, and F1-score in FIG. 9C, in which metrics are shown as functions of varying temporal restrictions on the trajectories. The top row provides a zoomed-in view of the results for shorter time windows (e.g., less than 30 seconds), while the bottom row displays the complete range of observation windows. FIGS. 10A-10C show mediumthreat prediction (MTP) settings, such as Precision in FIG. 10A, Recall in FIG. 10B, and F1-score in FIG. 10C, where metrics are shown as functions of varying temporal restrictions on the trajectories. The top row provides a zoomed-in view of the results for shorter time windows (e.g., less than 30 seconds), while the bottom row displays the complete range of observation windows.

[0142]

[0101] In the LTP scenario, late fusion demonstrates superior performance compared to the 11 classifiers evaluated, achieving high precision, recall, and F1-score across various observation windows. Due to the lower complexity of this setting, high performance is observed early in the trajectories, with the F1-score exceeding 0.85 after just a 10-second observation window.

[0143]

[0102] In the MTP scenario, the performance of DEWS exhibits a similar pattern, with late fusion consistently yielding better results. Precision and recall metrics suggest a well- calibrated system, with precision reaching approximately 0.95 after a 10-second observation window. Nonetheless, recall is considerably lower compared to the LTP setting, reflecting the increased challenge of identifying threatening trajectories in this more complex scenario.

[0144]

[0103] One or more aspects discussed herein can be embodied in computer-usable or readable data and / or computer-executable instructions, such as in one or more program modules, executed by one or more computers or other devices as described herein. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types whenAttorney Docket No. 009043.00129\WO NU2024-236-02 executed by a processor in a computer or other device. The modules can be written in a source code programming language that is subsequently compiled for execution or can be written in a scripting language such as (but not limited to) HTML or XML. The computer executable instructions can be stored on a computer readable medium such as a hard disk, optical disk, removable storage media, solid-state memory, RAM, and the like. As will be appreciated by one of skill in the art, the functionality of the program modules can be combined or distributed as desired in various embodiments. In addition, the functionality can be embodied in whole or in part in firmware or hardware equivalents such as integrated circuits, field programmable gate arrays (FPGA), and the like. Particular data structures can be used to implement one or more aspects more effectively, as discussed herein, and such data structures are contemplated within the scope of computer executable instructions and computer-usable data described herein. Various aspects discussed herein can be embodied as a method, a computing device, a system, and / or a computer program product.

[0145]

[0104] Although the present invention has been described in certain specific aspects, many additional modifications and variations would be apparent to those skilled in the art. In particular, any of the various processes described above can be performed in alternative sequences and / or in parallel (on different computing devices) to achieve similar results in a manner that is more appropriate to the requirements of a specific application. It is therefore to be understood that the present invention can be practiced otherwise than specifically described without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

[0146] REFERENCES:

[0147]

[0105] [1] A. Almohammad and A. Speckhard, “Isis drones: Evolution, leadership, bases, operations and logistics,’’ The International Center for the Study of Violent Extremism, vol. 5, 2017.

[0148]

[0106] [2] O. Ş EN and H. AKARSLAN, “Terrorist use of un- manned aerial vehicles:

[0149] Turkey’s example.’’ Defense Against Terrorism Review, vol. 13, 2020.Attorney Docket No. 009043.00129\WO NU2024-236-02

[0150]

[0107] [3] K. L. Best, J. Schmid, S. Tierney, J. Awan, N. M. Beyene, M. A. Holliday, R. Khan, and K. Lee, How to Analyze the Cyber Threat from Drones: Background, Analysis Frameworks, and Analysis Tools. Santa Monica, CA: RAND Corporation, 2020.

[0151]

[0108] [4] F. Ullah, S. M. Sepasgozar, and C. Wang, “A systematic review of smart real estate technology: Drivers of, and barriers to, the use of digital disruptive technologies and online platforms,” Sustainability, vol. 10, no. 9, p. 3142, 2018.

[0152]

[0109] [5] M. Thompson, A. A. Tarr, J. -A. Tarr, and S. Ritter- band, “Unmanned aerial vehicles: Liability and insurance,” in The Global Insurance Market and Change. Informa Law from Routledge, 2024, pp. 212-245.

[0153]

[0110] [6] C.-W. Wu, M.-D. Shieh, J.-J. J. Lien, J.-F. Yang, W.-T. Chu, T.-H. Huang, H.-C.

[0154] Hsieh, H.-T. Chiu, K.-C. Tu, Y.-T. Chen, S.-Y. Lin, J.-J. Hu, C.-H. Lin, and C.-S. Jheng, “Enhancing fan engagement in a 5g stadium with ai-based technologies and live streaming,” IEEE Systems Journal, vol. 16, no. 4, pp. 6590-6601, 2022.

[0155]

[0111] [7] V. Kulkami, A. Moro, and B. Garbinato, “Mobidict: A mobility prediction system leveraging realtime location data streams,” in Proceedings of the 7th ACM SIGSPATIAL International Workshop on GeoStreaming, 2016, pp. 1-10.

[0156]

[0112] [8] M. Raivi, S. A. Huda, M. M. Alam, and S. Moh, “Drone routing for drone-based delivery systems: A review of trajectory planning, charging, and security,” Sensors, vol.

[0157] 23, no. 3, p. 1463, 2023.

[0158]

[0113] [9] H. Georgiou, S. Karagiorgou, Y. Kontoulis, N. Pelekis, P. Petrou, D. Scarlatti, and Y. Theodoridis, “Moving objects analytics: Survey on future location & trajectory prediction methods,” arXiv preprint arXiv: 1807.04639, 2018.

[0159]

[0114]

[0010] Poibrenski, M. Klusch, I. Vozniak, and C. Muller, “M2p3: multimodal multipedestrian path prediction by self-driving cars with egocentric vision,” in Proceedings of the 35th Annual ACM Symposium on Applied Computing, 2020, pp. 190-197.

[0160]

[0115]

[0011] F. Chen, X. Wang, Y. Zhao, S. Lv, and X. Niu, “Visual object tracking: A survey,” Computer Vision and Image Understanding, vol. 222, p. 103508, 2022.Attorney Docket No. 009043.00129\WO NU2024-236-02

[0161]

[0116]

[0012] V. Bharilya and N. Kumar, “Machine learning for autonomous vehicle's trajectory prediction: A comprehensive survey, challenges, and future research directions,’’ Vehicular Communications, p. 100733, 2024.

[0162]

[0117]

[0013] L. Lin, W. Li, H. Bi, and L. Qin, “Vehicle trajectory prediction using lstms with spatial-temporal attention mechanisms,” IEEE Intelligent Transportation Systems Magazine, vol. 14, no. 2, pp. 197-208, 2021.

[0163]

[0118]

[0014] K. Messaoud, I. Yahiaoui, A. Verroust-Blondet, and F. Nashashibi, “Attention based vehicle trajectory prediction,” IEEE Transactions on Intelligent Vehicles, vol. 6, no. 1, pp. 175-185, 2020.

[0164]

[0119]

[0015] Y. Ren, Z. Lan, L. Liu, and H. Yu, “Emsin: enhanced multi-stream interaction network for vehicle trajectory prediction,” IEEE Transactions on Fuzzy Systems, 2024.

[0165]

[0120]

[0016] J.-H. Kim and D.-S. Kum, “Threat prediction algorithm based on local path candidates and surrounding vehicle trajectory predictions for automated driving vehicles,” in 2015 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2015, pp. 1220- 1225.

[0166]

[0121]

[0017] R. Bonatti, W. Wang, C. Ho, A. Ahuja, M. Gschwindt, E. Camci, E. Kayacan, S.

[0167] Choudhury, and S. Scherer, “Autonomous aerial cinematography in unstructured environments with learned artistic decision-making,” Journal of Field Robotics, vol. 37, no. 4, pp. 606-641, 2020.

[0168]

[0122]

[0018] V. Ajith and K. Jolly, “Unmanned aerial systems in search and rescue applications with their path planning: a review,” in Journal of Physics: Conference Series, vol.

[0169] 2115, no. 1. IOP Publishing, 2021, p. 012020.

[0170]

[0123]

[0019] M. Ozer, I. Keles, I. H. Toroslu, P. Karagoz, and S. Ergut, “Predicting the next location change and time of change for mobile phone users,” in proceedings of the third ACM SIGSPATIAL international workshop on mobile geographic information systems, 2014, pp. 51-59.

[0171]

[0124]

[0020] M. Ozer, I. Keles, I. H. Toroslu, and P. Karagoz, “Predicting the change of location of mobile phone users,” in Proceedings of the Second ACM SIGSPATIAL International Workshop on Mobile Geographic Information Systems, 2013, pp. 43-50.

Claims

Attorney Docket No. 009043.00129\WO NU2024-236-02What is claimed is:

1. A method comprising:receiving, from a drone tracking system, a drone trajectory;extracting, by a feature extraction module, drone trajectory features from an observed portion of the drone trajectory, wherein the drone trajectory features comprise drone positioning information and geolocated asset information;classifying, by a threat classification model and based on the drone trajectory features, the drone trajectory with a predicted threat classification comprising a plurality of threat predictions, wherein the extracting and the classifying are performed in approximately 3 seconds, enabling real-time threat prediction;combining, by a fusion module, the plurality of threat predictions into a final threat classification; andtriggering, when the final threat classification meets a threshold, a threat assessment alert.

2. The method of claim 1, comprising training, based on historical drone trajectory information captured from historical drone flights, one or more machine learning models of the threat classification module, wherein the historical drone trajectory information comprises self-similarity features representing similarity to past trajectories of a same drone and crosssimilarity features representing similarity to past trajectories of other drones.

3. The method of claim 2, the drone trajectory information comprises annotated information corresponding to classified geolocated features of a municipality.

4. The method of claim 1, wherein the geolocated asset information comprises values assigned to assets in the area over which the drone trajectory passes.

5. The method of claim 1, wherein the classifying is performed based on a suite of machine learning classifiers.

6. The method of claim 1, wherein the classifying comprises combining a threat prediction from a plurality of machine learning classifiers.Attorney Docket No. 009043.00129\WO NU2024-236-027. The method of claim 6, wherein the combining comprises a weighted sum of probability estimates of a plurality of predictions from the plurality of machine learning classifiers.

8. The method of claim 1, comprising continually providing forecasted threat assessments based on 5 seconds of an observed drone trajectory.

9. The method of claim 8, wherein a threat assessment of the forecasted threat assessments is tailored to a specific context of a drone flight.

10. The method of claim 9, wherein the specific context and threat assessment comprises one or more of a particular threat assessment, a particular asset, and a capability of an identified drone.

11. The method of claim 1, wherein the observed portion of the drone trajectory comprises trajectory data captured within an observation window of 30 seconds or less from a start of a drone flight.

12. A system comprising:a drone tracking system monitoring communications at communication frequencies utilized by drone communications with drone operatorsa processor; anda memory storing instructions that, when executed by the processor, cause the system to:receive, from the drone tracking system, a live drone trajectory; extract drone trajectory features from live trajectory information of the live drone trajectory, wherein the drone trajectory features comprise at least asset-based features representing values of geolocated assets over which a drone trajectory passes;generate, using a suite of machine learning classifiers, a plurality of threat predictions based on the drone trajectory features;combine, by a fusion module, the plurality of threat predictions to produce a final threat score; andoutput a threat assessment based on the final threat score.Attorney Docket No. 009043.00129\WO NU2024-236-02 13. The system of claim 12, wherein the instructions further cause the system to extract the drone trajectory features from an initial portion of the drone trajectory captured within an observation window, and wherein the observation window comprises one of 5 seconds, 10 seconds, 20 seconds, or 30 seconds from a start of a drone flight.

14. The system of claim 12, wherein the suite of machine learning classifiers comprises eleven classifiers including one or more of Logistic Regression, k-Nearest Neighbors, Support Vector Machines, Decision Trees, Random Forest, Gradient Boosting, Naive Bayes, AdaBoost, Extra Trees, a Multi-layer Perceptron, and a wide-and-deep neural network.

15. The system of claim 12, wherein the drone trajectory features further comprise one or more of basic features, capability features, altitude features, speed features, no-fly zone features, and historical observation features.

16. The system of claim 12, wherein the instructions further cause the system to continually provide forecasted threat assessments based on observed drone trajectory data starting from 5 seconds of observation.

17. The system of claim 12, further comprising an annotation tool comprising a user interface configured to receive user input associating geolocated assets with value classifications for training the suite of machine learning classifiers.

18. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:receive trajectory data corresponding to an observed portion of a drone flight; extract a plurality of features from the trajectory data, the plurality of features comprising asset-based features representing values of geolocated assets over which the drone flight passes;apply a plurality of trained machine learning classifiers to the plurality of features to generate a plurality of threat predictions;combine the plurality of threat predictions using late fusion to generate a final threat classification; andAttorney Docket No. 009043.00129\WO NU2024-236-02 generate an alert when the final threat classification exceeds a predetermined threshold.

19. The non-transitory computer-readable medium of claim 18, wherein the observed portion of the drone flight comprises trajectory data captured within 30 seconds of a start of the drone flight, and wherein the final threat classification achieves an F1-score of at least 0.8.

20. The non-transitory computer-readable medium of claim 18, wherein the assetbased features are weighted as a primary determinant in generating the final threat classification relative to other features of the plurality of features.

Images