Explainable ai (XAI) for object detection and application to satellite imagery
By defining meaningful object parts and verifying their presence before prediction, the model enhances transparency and robustness against adversarial attacks, addressing the lack of explainability and vulnerability in deep learning models for satellite imagery.
Patent Information
- Application Number
- PCT/US2026/026970
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-06
- Filing Date
- 2026-05-06
- Publication Date
- 2026-11-12
AI Technical Summary
Deep learning models for object detection in satellite imagery lack transparency and are vulnerable to adversarial attacks, necessitating a solution that ensures explainability and robustness.
A user-centric approach where users define meaningful parts of objects, and the model is trained to verify these parts before making predictions, incorporating a post-processing phase to ensure part-based verification, enhancing explainability and resilience against adversarial attacks.
The approach achieves higher accuracy and robustness against adversarial attacks, ensuring compliance with regulatory requirements for transparency and improving decision-making in applications like satellite imagery analysis.
Smart Images

Figure US2026026970_12112026_PF_FP_ABST
Abstract
Description
Attorney Docket No.: 37684.6115WOM25-312P-WO1-bEXPLAINABLE Al (XAI) FOR OBJECT DETECTION AND APPLICATION TO SATELLITE IMAGERYCLAIM OF PRIORITY
[0001] This application claims the benefit of U. S. Provisional Patent Application No.63 / 801,003, filed May 6, 2025, entitled “EXPLAINABLE Al (XAI) FOR OBJECT DETECTION AND APPLICATION TO SATELLITE IMAGERY”, the disclosure of which is incorporated by reference herein in its entirety.COPYRIGHT NOTICE
[0002] A portion of this document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the document as it appears in the Patent and Trademark Office records but otherwise reserves all copyright rights whatsoever.TECHNICAL FIELD
[0003] The disclosed embodiments relate to a transparent and / or an explainable Al (XAI) model for object detection.BACKGROUND
[0004] DARPA has focused attention on model explainability because deep learning Al models were too opaque and hard to understand and trust. DARPA started funding research on explainability through its Explainable Al (XAI) program in 2015 (Explainable Artificial Intelligence (darpa.mil)). That concern and the danger of using such Al systems led the EU parliament to pass the European Al Act on March 13, 2024 (The Act Texts | EU Artificial Intelligence Act), which is a comprehensive attempt to regulate artificial intelligence (Al) globally. The EU Al Act entered into force on August 1, 2024. One of the primary provisions in the Act is about explainability.
[0005] There are other articles in the Al Act related to explainability and transparency. The focus of the Al Act is on high-risk Al systems and many of the computer vision and object detection systems that use deep learning models - such as those used in autonomous vehicles, video surveillance, medical diagnosis and various robotic systems -conceivably fall under the high-risk category.Attorney Docket No.: 37684.6115WOM25-312P-WO1-bBRIEF DESCRIPTION OF THE DRAWINGS
[0006] Embodiments are illustrated by way of example, and not by way of limitation, and can be more fully understood with reference to the following detailed description when considered in connection with the figures in which:
[0007] Figure 1 presents DARPA’s vision of Explainable Al.
[0008] Figure 2 illustrates annotations for a destroyer (on the left) and a cruiser (on the right) in a satellite image such as conning towers / bridges (control towers), helipads (helicopter pads), guns, and missile launchers.
[0009] Figure 3 shows features of two aircraft carriers in a satellite image identified by the XAI model in accordance with DARPA-style explanation.
[0010] Figure 4 illustrates an XAI model that correctly identified the launchpad of the aircraft carrier on the right even though a patch attack covered up a large portion of it. However, the XAI model failed to identify any portion of the launchpad of the aircraft carrier on the left with a reasonable level of confidence, even though it has high confidence (0.89) that it’s an aircraft carrier.
[0011] Figure 5 illustrates a camouflage attack on three ships (the right aircraft carrier and two ships furthermost on the right) with Attack Strength = 0.25.
[0012] Figure 6 illustrates a camouflage attack on three ships (the right aircraft carrier and two ships furthermost on the right) with Attack Strength = 0.5.
[0013] Figure 7 illustrates a camouflage attack on three ships (the right aircraft carrier and two ships furthermost on the right) with Attack Strength = 0.75.
[0014] Figure 8 illustrates parent class detections contesting for the same region, according to the disclosed embodiments.
[0015] Figure 9 illustrates resolving a contest between parent class detections over the same region, according to the disclosed embodiments.
[0016] Figure 10 illustrates dropping losing parent and child class detections, according to the disclosed embodiments.Attorney Docket No.: 37684.6115WOM25-312P-WO1-b
[0017] Figure 11 illustrates adding low-confidence child detections to high confidence parent class detections, according to the disclosed embodiments.
[0018] Figure 12 illustrates adding low-confidence child detections to high-confidence parent class detections, according to the disclosed embodiments.
[0019] Figure 13 presents Table 1 which shows the “Other Ships (yachts, motorboats)” that are part of the “Merchant” class in the ShipRSImageNet dataset.
[0020] Figure 14 presents Table 2 which shows the parts defined for three merchant ships and three warship classes.
[0021] Figure 15 presents Table 3 which shows that the Warships and Merchant Ships categories contain some classes of ships for which parts are not defined in the disclosed embodiments.
[0022] Figure 16 presents Table 4 which shows the total number of objects used to train and test XAI and non-XAI YOLO models.
[0023] Figure 17 presents Table 5 which shows the results for XAI and non-XAI models with increasing size of the patches.
[0024] Figure 18 presents Table 6 which shows an experiment testing the disclosed embodiments in which Attack Strength is varied from 25% to 75%, as discussed further below.
[0025] Figure 19 presents Table 7 which shows the counts for each type of plane in experiments that trained models to identify three types of airplanes - fighter, transport, and civil planes, according to the disclosed embodiments.
[0026] Figure 20 presents Table 8 which shows the results for XAI and non-XAI models with increasing size of the patches.
[0027] Figure 21 presents Table 9 which shows the average and standard deviations of the various measures over four folds for XAI and non-XAI models for the three types of planes.
[0028] Figure 22 presents Table 10 which shows the object count distribution for the ship detection dataset, according to the disclosed embodiments.Attorney Docket No.: 37684.6115WOM25-312P-WO1-b
[0029] Figure 23 presents Table 11 which shows the sample four-fold results from which were computed the averages and standard deviations of the various measures, according to the disclosed embodiments.
[0030] Figures 24A-24F provide some examples of patch attacks showing the advantage of XAI models according to the disclosed embodiment compared to non-XAI models.
[0031] Figures 25A and 25B illustrate examples of camouflage attacks showing the advantage of XAI models compared to non-XAI models.
[0032] Figures 26A-26D provide some examples of patch attacks showing the advantage of XAI models compared to non-XAI models.
[0033] Figures 27A-27D provide some examples of camouflage attacks showing the advantage of XAI models compared to non-XAI models.
[0034] Figure 28 provides a flowchart according to the disclosed embodiments.DETAILED DESCRIPTION
[0035] Explainability, in terms of Artificial Intelligence (Al) models, is critical to wider deployment of deep learning Al models. This is especially true in view of the worldwide effort to regulate Al with laws that impose a requirement for transparency in Al systems. The disclosed embodiments present a new explainable modeling approach for object detection that generally applies to classification problems as well. In this modeling approach, prediction of an object in an image depends on explicit verification of some of its parts as defined by users. For example, the fuselage, tail, wings and engines are airplane parts that may need to be verified. Identification of meaningful parts of objects by users is at the core of this form of explainability. User involvement in parts identification implies that these parts have meaning to them. In the disclosed embodiments, the process of creating explainable models is reversed. The approach of the disclosed embodiments is like an architect first consulting with a client to determine the needs before designing and building a house. Thus, the definition of explanation comes first, then comes model creation based on that definition. The US defense agency DARPA originally proposed this type of explicit, symbolic form of explainability when it started the Explainable Al (XAI) program in 2015. The disclosed embodiments apply the DARPA idea of XAI to detect objects in satellite imagery. The XAI object detection models according to the disclosed embodiments are tested on some publicly available satellite imagery to find ships andAttorney Docket No.: 37684.6115WOM25-312P-WO1-bairplanes. The disclosed embodiments show that these explainable models can provide higher accuracy and a higher level of protection against adversarial attacks.1. INTRODUCTION
[0036] The EU Al Act entered into force on August 1, 2024, which means Al is now regulated in the EU. One of the primary provisions in the Act is about explainability. Article 13 (1) of the Act states the following:High-risk Al systems shall be designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users to interpret the system ’s output and use it appropriately. An appropriate type and degree of transparency shall be ensured, with a view to achieving compliance with the relevant obligations of the user and of the provider set out in Chapter 3 of this Title.
[0037] There are other articles in the Al Act related to explainability and transparency. The focus of the Al Act is on high-risk Al systems and computer vision and object detection systems that use deep learning models - such as those used in autonomous vehicles, video surveillance, medical diagnosis and various robotic systems. These systems conceivably fall under the high-risk category. Walke, F., Bennek, L., & Winkler, T. J. (2023), Artificial Intelligence Explainability Requirements of the Al Act and Metrics for Measuring Compliance, aisel.aIsnet.org / wi2023 / 77 / provides a comprehensive analysis of the explainability provisions of the Al Act and concludes that the explainability requirement is binding. Pavlidis, G. (2024), Unlocking the Black Box: Analyzing the EU Artificial Intelligence Act’ s Framework for Explainability in Al, Law, Innovation and Technology, 16(V), 293-308 states that the “notion of explainability is one of the fundamental principles that underpin the Al Act. ”
[0038] Given this mandate for XAI in the EU Al Act, a form of explanation is illustrated in FIG. 1. The form or explanation was originally proposed by DARPA (Defense Advanced Projects Agency, US) in its Explainable Al (XAI) program. As shown in the “Tomorrow” part of FIG. 1, to predict that there is a cat in the image, this form of explanation requires the model to additionally verify the existence of some of its unique parts - such as fur, whiskers, and claws - and provide such verified information as model output. In general, this form of explanation implies that an object consists of parts and that the model not only needs to verify internally the existence of some of its unique parts but also output that information in an explicit symbolic form. Here the cat, fur, whiskers, and clawsAttorney Docket No.: 37684.6115WOM25-312P-WO1-bare abstract concepts and the symbolic logic, applied to the model outputs to predict that it’s a cat, is like this: IF the Fur is that of a Cat AND the Whiskers are that of a Cat AND the Claws are that of a Cat THEN It’s a Cat.
[0039] Among the potential benefits of this type of XAI model is that it can produce detailed information about objects and, when integrated with contextual information, it can lead to better decision-making. For example, such an object detection system, when applied to satellite imagery during warfare, can provide better targeting information - such as identifying the location of the control tower of an aircraft carrier or a destroyer. Another benefit is continuously recording explanatory information in automated processes such as high-speed assembly lines and surveillance systems. Having explanatory information recorded continuously can facilitate investigations, for example, when things go wrong.
[0040] Of course, one benefit is compliance with the EU Al Act from the start of the process of model building. Thus, in the first step of the disclosed embodiments, the user defines the form of the explanation. For example, in detecting airplanes in satellite imagery, the end user defines the parts of different types of aircraft. In this case, one user might specify the parts of an airplane as consisting of the wings, tail and fuselage and nothing more. Another user might add aircraft engines and the cockpit to that list. Additionally, or alternatively, one user might specify a particular form or shape of one or more parts of the airplane, and another user might specify a different form or shape of one or more parts of the airplane. Thus, in the disclosed embodiments, each user is in complete control of the form of the explanation, which in this case is making a list of the parts of objects that need verification. In other words, the user is responsible for making the model transparent to the extent needed for the relevant application.
[0041] Thus, the disclosed embodiments present a new method to create an explainable object detection model that exploits the explanatory concept outlined by DARPA. Note that this form of explanation is also related to the idea of causal models and concept extraction methods.2. METHOD
[0042] The disclosed embodiments were tested on publicly available satellite imagery to find ships and airplanes based on their parts and demonstrated that this part-based approach to object detection generally produces higher accuracies compared to standard object detection models that doAttorney Docket No.: 37684.6115WOM25-312P-WO1-bnot have parts defined for them. The parts-based object detection models according to the disclosed embodiments were also found to be generally more resilient to adversarial attacks.
[0043] In most of the current approaches to model explanation, complex models are built first and explainability is added afterward using techniques such as saliency maps and grad-CAMS. The disclosed embodiments reverse this process. Explainability requirements are first defined and then appropriate complex models are built that adhere to those specifications. This ensures that the models are inherently explainable, rather than requiring separate, post-hoc explanation methods. The process is analogous to an architect meeting with clients to determine their needs before designing and constructing a house. The modeling approach in the disclosed embodiments is fundamentally user centric.3. RELATED WORK
[0044] This section discusses some general ways to classify XAI methods that relate to the disclosed embodiments.
[0045] XAI methods and models are classified in the following ways:
[0046] (1) Intrinsic vs. Post-Hoc Methods: A constructed model that is inherently interpretable is referred to as an intrinsic XAI method. Decision trees and linear models are generally considered inherently interpretable. A method of explainability that is applied to a complex model after it is built is referred to as a post-hoc XAI method. Thus, a post-hoc XAI method is external to the underlying model. A post-hoc method generally tries to explain an underlying complex model either by constructing another interpretable surrogate model using the complex model’s inputs and outputs, or by simply explaining individual predictions using other methods such as a heat map or saliency map. The XAI method presented in the disclosed embodiments, that uses the DARPA concept, has an intrinsic explainable part that is built into the output layer of an object detection model. DARPA calls it the Explanation Interface in FIG. 1.
[0047] (2) Global vs. Local Methods: An XAI method that provides global explanations -such as the DARPA model that predicts a cat based on its parts with the basic logic applying to all cats in all images - is referred to as a global method. A method that uses a mode of explanation that only works on a case-by-case basis - such as in a visualization method that highlights the set of pixels where a cat is, but that set of cat pixels changes from one image to another - is referred to as a local method. Sensitivity maps, saliency maps, variations of grad-cams and similar pixel identification (or visualization) methods are all local methods. Global methods, such as the DARPA one, can produceAttorney Docket No.: 37684.6115WOM25-312P-WO1-bbroad generalizations to understand a phenomenon unlike local methods. Local methods do not have the generalization capability. However, the DARPA concept can be modified to work as a local model too. For example, an image might have only the face of a cat and no other cat parts are visible. Just like any human, a slightly modified DARPA model should still predict that there is a cat in the image based on that one visible part. In other words, as long as at least one part of an object is visible, the part-based prediction method should still work. Thus, the DARPA model can be both global and local.
[0048] (3) Model-agnostic vs. Model-specific: The model-agnostic vs. model-specific designation applies to post-hoc methods only because intrinsic models are model-specific anyway. Model agnostic means that the post-hoc method applies to any underlying machine learning (ML) model, whereas model-specific means it only applies to certain ML models. There is a wide range of methods that are model agnostic, from visualization methods and surrogate models (e.g., interpretable models, such as decision trees, built from predictions of complex models) to sensitivity analysis (e.g., testing solution stability by changing weights) and example-based explanation (e.g., counter factual explanations). The XAI method according to the disclosed embodiments can work with a wide range of object detection models and, therefore, is model agnostic.4. OBJECT DETECTION ALGORITHMS
[0049] Object detection algorithms provide three outputs: bounding boxes, objectness, and category or class. Bounding boxes provide the location of objects in an image, objectness provides the confidence that a located area has an object, and category provides the class of the object. Major object detection algorithms are either single stage or two-stage. Two-stage algorithms first provide region proposals and then check for objects in those regions. Major two-stage algorithms include regions with CNN (R-CNN), Fast R-CNN, Faster R-CNN, and Mask R-CNN. Two-stage detectors usually have higher accuracy but are slower. One-stage or single-shot algorithms skip the region proposal generation phase and do both classification and bounding-box regression directly in one step. Major one-stage algorithms include You Only Look Once (YOLO), single-shot detector (SSD), and RetinaNet.
[0050] Satellite imagery was used to experiment with the XAI method in the disclosed embodiments, and object detection in satellite imagery presents many challenges because of class variations, and high variance in object size, illumination and background complexity. Small objects are generally hard to locate. In addition, object density and occlusion affect the detection performance.Attorney Docket No.: 37684.6115WOM25-312P-WO1-b5. ADVERSARIAL ATTACKS IN IMAGE RECOGNITION ANDOBJECT DETECTION
[0051] Since the method in the disclosed embodiments is tested with adversarial attacks, this section reviews some adversarial attack methods. Despite recent advances, deep learning-based object detection models remain vulnerable to adversarial attacks. Adversarial attacks introduce perturbations into input images to cause object detection models to fail either through wrong object predictions and / or not be able to recognize some objects in the images. This behavior occurs in DNN-based models. In general, there are different ways to classify adversarial attacks. In digital type attacks, perturbations are in the digital space. Physical type attacks correspond to real world on-object perturbations, commonly achieved by affixing physical artifacts - such as stickers, posters, and clothes - on objects in the physical world. Knowledge of the DNN model is another way to classify adversarial attacks. In White-Box attacks, the adversary has full knowledge of the model, including its architecture, parameters, and weights. In Black-Box attacks, the adversary has no knowledge of the model’s structure and parameters. In Gray-Box attacks, the adversary has partial knowledge of the model, such as the model’s structure or the output probability distribution. Adversarial attacks can also be categorized based on their goals, whether its Targeted or Untargeted. In Untargeted attacks, the goal of the attack is simply to produce a wrong prediction, not any specific wrong prediction. In Targeted attacks, the goal is to produce a specific wrong prediction, such as a cat to be predicted as a dog.A. GENERAL PHYSICAL ADVERSARIAL ATTACKS ON IMAGE RECOGNITION
[0052] Many physical adversarial attack experiments create adversarial images in the digital space first and then print them out for physical attacks. An optimization framework, called Expectation Over Transformation (EOT), can construct robust adversarial examples for physical attacks. GAN may be used to generate adversarial examples. These are all white-box attacks.B. PHYSICAL ADVERSARIAL A TTA CKS ON OBJECT DETECTION
[0053] The main techniques for physical adversarial attacks on object detection consists of adversarial patches, adversarial camouflage, and other approaches, such as emitting laser lights towards objects.C. PATCH-TYPE PHYSICAL ADVERSARIAL ATTACKS
[0054] Others in the prior art have experimented with adversarial attacks on object detectors by modifying the texture on stop signs. Still others in the prior art used the EOT framework to create robust adversarial patches for two-stage object detectors. Others in the prior art presented techniquesAttorney Docket No.: 37684.6115WOM25-312P-WO1-bfor creating robust physical adversarial examples and road-tested them against a version of YOLO and Faster RCNN. Still others in the prior art introduced the idea of creating adversarial patches that can be placed anywhere in the image, in essence a location invariant patch. Some in the prior art attached their adversarial patches to the camera lens and fooled a version of YOLO and Faster RCNN about a third of the time. Some in the prior art pasted their adversarial patches on toy car hoods and achieved a very high (greater than 90%) success rate. These prior art techniques digitally created the adversarial patch first and then printed out the patch for the attack. Some in the prior art have suggested that these prior art techniques are only suitable for static scenarios and proposed creating dynamic adversarial patches and displaying them with a digital screen. Others in the prior art proposed a general framework to construct robust adversarial patches. Given that small perturbations may not be able to deceive models equipped with adversarial defenses, they use large perturbations to break adversarial defenses. Still others in the prior art proposed a method to attack automatic check-out systems in shopping centers with a universal adversarial patch. They printed the adversarial patches and pasted them on the shopping items for physical attacks. Their experiments show their attack method could degrade recognition accuracy by about 40%.
[0055] Some in the prior art created adversarial patches for non-rigid deformable objects (i.e., persons) and demonstrated that the generated patches could hide the persons in most cases for a YOLO model. Others in the prior art created adversarial patches for T-shirts to deceive pedestrian object detectors. Still others in the prior art constructed universal adversarial patches that are imperceptible. In a two-stage process, some in the prior art optimized the adversarial patches such that they are imperceptible to humans and can also evade object detectors. These are all white-box methods.D. CAMOUFLAGE-TYPE PHYSICAL ADVERSARIAL ATTACKS
[0056] Camouflage generally involves changing the color or texture of an object to match the surroundings. Camouflage is standard in warfare with the military trying to hide equipment and personnel by covering them with vegetation, nets of various types, and clothing. With the use of satellites and drones in warfare, militaries have started to paint warships, sometimes with different colors for different parts, so that they are hard to detect. This can hide the overall form and length of a warship.
[0057] ShipCamou is a type of adaptive patch method for physical camouflage of a ship. For white-box attacks, ShipCamou optimized the patches so that the object detection algorithm produces distorted bounding boxes and lower confidence levels. ShipCamou was tested on the HRSC-2016Attorney Docket No.: 37684.6115WOM25-312P-WO1-bmaritime remote sensing image dataset against a variety of object detection algorithms in both whitebox and black-box mode.
[0058] There are those in the prior art that argue that existing physical attack methods cover large areas and are easily discernable by humans, and present as an alternative a method to generate natural patches that are small and imperceptible to humans. The alternative method was tested on YOLOv3 using multiple datasets and achieved high success rate in both the digital and physical domain.
[0059] Others in the prior art proposed CamoNet, a target camouflage network, that can create imperceptible perturbations. CamoNet was tested on three object detection algorithms - RetinaNet, Faster R-CNN, and RTMDet - using the DOTA and DIOR satellite imagery datasets.E. ADVERSARIAL ATTACK ON AERIAL SURVEILLANCE SYSTEMS
[0060] The use of object detection systems in aerial surveillance systems (drones, UAVs) has increased in recent years. The use of camouflage nets to hide military hardware is quite common. Recent studies show that even partial imagery patches can deceive aerial surveillance systems. The impact of the patch size, position, number, and saliency on adversarial performance has been investigated on a YOLOv2 model trained on aerial imagery. The investigation used a unique loss function to optimize the patches and showed that optimized patches could reduce the YOLOv2 performance by 47%. Others in the prior art devised scale- adaptive patches that were placed on top of the target objects. Scale-adaptiveness was achieved by controlling the ratio of the patch size and the detected bounding box. In simulation experiments with an airport sandbox and using optimized patches, the detection performance of the object detector degraded substantially.6. AN INTUITIVE OVERVIEW OF THE ALGORITHM ACCORDING TO THE DISCLOSED EMBODIMENTA. PARTS ARE DEFINED B Y USERS
[0061] The form of explainability according to the disclosed embodiments starts with users defining parts of objects that are meaningful to them. The parts are essentially subclasses of the main classes. Unlike other XAI methods, the disclosed embodiments start with the form of explanation that users want and creates a corresponding object detection model. The object detection model is trained to detect both the objects and their parts. For example, one can say that the main distinction between cats and dogs is in the head and that is the only part that an XAI model should verify. In this case, theAttorney Docket No.: 37684.6115WOM25-312P-WO1-bdisclosed embodiments define and annotate four classes for the object detection model - cat, cat head, dog, and dog head. In another situation, suppose that one would like to verify at least two parts out of three and therefore defines and annotates three parts of cats and dogs; suppose they are the head, tail and legs. In this case, the object detection model has eight classes, four each for cats and dogs. In each of these cases, the object detection model is different because the set of classes is different. This is a top-down process for XAI model development where one defines the form of the explanation first and then trains the appropriate object detection model from the given form of explanation, which is a list of meaningful parts to verify.B. ANNOTATE PARTS OF OBJECTS, BUILD OBJECT DETECTION MODEL
[0062] Image annotation for object detection begins once the users define the form of the explanation in terms of identifying meaningful parts of objects. FIG. 2 shows the annotations for a destroyer (on the left) and a cruiser (on the right) in a satellite image where the defined parts are a conning tower / bridge (control tower), a helipad (helicopter pad), a gun, and a missile launcher. FIG. 3 shows how the XAI object detection model, trained to detect aircraft carriers (and other kinds of ships and vessels) and their parts in satellite images, locates and identifies the parts with bounding boxes. The figure shows two aircraft carriers, and both are identified with their parts. Bounding boxes 300 contain the control towers, bounding boxes 305 contain the runways, and bounding boxes 310 contain the launchpads. Bounding boxes 315 contain the whole object - the aircraft carriers. The numbers associated with the bounding boxes correspond to a confidence level or confidence score of the respective identifications, predictions, and / or locations.C. POST-PROCESSING - OBJECT PREDICTION CONTINGENT ON PART VERIFICATION WITH SOME EXCEPTIONS
[0063] Object parts serve a purpose beyond providing explanation. In addition to explanation, the parts are used for object verification. In other words, before the disclosed embodiments predict that an object exists in an image, the disclosed embodiments verify that one or more parts of the object are visible. This verification is completed in a post-processing phase where the output of the object detection algorithm is analyzed. For example, with reference to FIG. 3, before the disclosed embodiments predict that there are two aircraft carriers in the image, the disclosed embodiments verify for each one that at least some of the parts - such as control tower, the runway, and the launchpad -were visible and found by the object detection algorithm.Attorney Docket No.: 37684.6115WOM25-312P-WO1-b
[0064] In general, because of the possibility of adversarial attacks, for any object prediction the disclosed embodiments ensure at least one of the parts of the object is detected. This means that if the disclosed embodiments do not detect any of the parts of an object, then it does not predict that the object exists in the image. Thus, at least one part is the minimum requirement for object prediction. One exception that may be made is when the confidence in the prediction is high, but the object detection algorithm did not find any of the parts of the object, or the parts had very low confidence. This post-processing logic is applied to each object detected by the object detection algorithm.
[0065] This post-processing logic for part verification-based prediction is what makes the method in the disclosed embodiments unique.D. SIMULATING PHYSICAL ADVERSARIAL ATTACKS FOR ADVERSARIAL TESTING - PATCH ATTACKS
[0066] In the disclosed embodiments, physical adversarial attacks were simulated, both patch and camouflage types. Patches of different shapes (e.g., round, square, rectangle) and sizes were created and placed at random positions on parts of objects. The sizes of the patches were gradually increased so that they almost covered the object parts. Patches independent of object detection models were created. Hence, the disclosed embodiments are a black-box method of adversarial attack. However, the patch attack method is targeted because the patches are placed on parts of objects so that the objects are misclassified or become invisible.
[0067] FIG. 4 shows different sized patches (square, rectangular and circular) 400 placed on different parts of two aircraft carriers identified by bounding boxes 315. The patches completely cover the control towers on both carriers. And the runways are also almost covered. The launchpads are partially covered. Despite such a patch attack, the XAI object detection model in accordance with the disclosed embodiments still identified the two aircraft carriers. For the carrier on the right, it still recognized launchpad 310 even though it is partially covered, showing that the part-based recognition worked. For the carrier on the left, the model could not recognize any of the parts 300, 305 and 310. An exception is made to the part-based recognition idea for such cases because the confidence level (0.89) is very high. These exceptions generally occur with adversarial attacks.E. SIMULATING PHYSICAL ADVERSARIAL ATTACKS FOR ADVERSARIAL TESTING - CAMOUFLAGE ATTACKS
[0068] One form of camouflage is where one paints or covers military equipment to make them blend in with their surroundings so that they become invisible. The disclosed embodiments simulate aAttorney Docket No.: 37684.6115WOM25-312P-WO1-bcamouflage-type adversarial attack by changing the color of the ships and airplanes to gradually blend with the color of the surroundings. FIGS. 5, 6 and 7 show this gradual change of color for three ships 500, 505 and 510 in the image. The color of the other two ships, 515 and 520, was changed to a lesser degree to show the color difference. The degree of color change is called “Attack Strength.” An Attack Strength = 0 means there was no color change. An Attack Strength = 1 means the ship’s color matches the surrounding water and becomes completely invisible. The Attack Strength varies, for example, from 0.25 to 0.75 in the disclosed embodiments.7. THE ALGORITHMIC DETAILS
[0069] An overview of the basic steps of the XAI algorithm according to the disclosed embodiments follows:1. Ask users to define parts of objects for explainability: Ask users what parts of objects require verification for object prediction and explainability. Make a list of such parts for each object.2. In the training dataset, annotate each object and the corresponding parts in each image.3. Train an object detection model with the training dataset where the objects and their corresponding parts are separate classes of objects.
[0070] The essence of the XAI object detection model according to the disclosed embodiments is in the post-processing logic used in the testing phase. In the post-processing logic, the disclosed embodiments verify that the appropriate parts of objects are found before any object prediction. Next are the algorithmic details of the post-processing phase for object prediction.Attorney Docket No.: 37684.6115WOM25-312P-WO1-bA. XAI POST PROCESSING ALGORITHM - PART VERIFICATION FOR OBJECT PREDICTION
[0071] Definitions:U = Set representing all classes detected by our model.P It is the set of main object classes, termed ParentClasses.C ⊂ U is the set of part object classes, termed ChildClasses.Additionally, these sets satisfy the condition: P ∩ C = ∅Let D be the set of all detections:T‘> ~ (ob | di ~ (Ci.b^si) }, (1)where:* c,- is the class label of detection d>.* bi represents the bounding box coordinates of* s_i is the confidence score of d_iOverlap Ratio.Area(b H )Area(bB. KNOWLEDGE BASE
[0072] Define an association mapping A from P to subsets of C such that each parent p E P is associated with a subset of child classes in C. Formally, the mapping can be represented as:A: P → 2^C (3)
[0073] Thus, for each parent class p E P, there exists a subset A(p) c C representing the child classes associated with p. This mapping is implemented as a dictionary where the keys are parents (Main Class) and values are their children (Parts).
[0074] Example Association Mapping:4 (Warship X) = {Control Tower, Gun, Helipad},4 (Warship Y) = {Control Tower, Launcher, Vertrep}Attorney Docket No.: 37684.6115WOM25-312P-WO1-bC. ALGORITHMStep 1: Initial Detection Filtering:
[0075] Define a minimum confidence threshold score T0such that detections with st> T0are retained:® = { dt| st> T0}. (4)
[0076] Initialize the final detection set as an empty set.T = { }. (5)Step 2: Categorize Detections into Disjoint Sets:
[0077] Define a higher (stable) confidence threshold i. This threshold can be chosen based on empirical evidence. Partition T> into four disjoint sets:Xp= {di E D | ctE P, Si > Ti}, (High-confidence parents)Xc= {di E D | Ci E C, st> T-L}, (High-confidence children)= {di E T> | Ci E P, T0< Sj < T-L}, (Low-confidence parents) Lc= {dLE T> | ctE C, T0< Sj < T-L). (Low-confidence children) (6)Step 3: Construct Overlap Graph G:
[0078] Define an overlap threshold 90. Construct an undirected graph.Step 4: Find Connected Components:
[0079] Find the sets of connected components X = {X X2, — > Kk} ingraph G. Each set of connected components Xkrepresents a group of parent detections contesting for the same region. That means they are overlapping detections that are on the same object. So, some of them will need to be dropped.
[0080] FIG. 8 showsT>2) and (T)3, T)^) are connected. After the graph is built, the following set of connected components is obtained: X = {{7)1, P>2}, {P3, T }}.Attorney Docket No.: 37684.6115WOM25-312P-WO1-bStep 5: Determine Winning Detections:
[0081] Define an overlap threshold 6, which is used to detect the presence of a child (part class) within the parent (main class).
[0082] Initialize an empty set of winners W
[0083] For each connected component Xk:1. For each detection d_i ∈ Xk, calculate the total confidence score:2. Select the detection d_i* ∈ Xkwith the highest S_id* = arg max S_i. (9)3. Add d_i* to the winners set W.
[0084] Consider FIG. 9 where each bounding box has a confidence score. The two large bounding boxes 900 enclosing the main object represent the parent class detections, while the smaller bounding boxes 905 correspond to child class detections.
[0085] The score calculation is as follows, adding the child confidence score to the parent confidence score:W = {}Kk= {G, R}Score (G) = 0.75 + (0.7 + 0.9 + 0.8)Score (R) = 0.8 + (0.3)
[0086] Since,Score (G) > Score (R)
[0087] The winner is G:W = {G}Attorney Docket No.: 37684.6115WOM25-312P-WO1-bStep 6: Remove Losers and Their Associated Parts:
[0088] FIG. 10 illustrates dropping losing parent and child class detections. With reference to FIG. 10, the disclosed embodiments update the Hpand Hcsets by removing losing parent detections and their associated children. Let Lpossdenote the set of parent detections that lose within each connected component Xk. This set is defined as:Lposs= ∪(Xk\{d*}) (10) where d* is the winning detection from the set of connected components Xk.
[0089] The updated set of high-confidence parent detections is given by:
[0090] For each losing parent detection dLG £pOSS, associated child detections dj Gcthat meet the following criteria are identified:Cj E A(c and O bj) > 0!, (12)
[0091] where A(c_i) denotes the associated child classes for the parent class c_i andis the overlap ratio between the parent dLand child dj.
[0092] The set of associated child detections to be removed is denoted as £e’ssand defined as:
[0093] The updated set of high-confidence children is given by:C=C\ "SS. (14)
[0094] Add detections to the final detections:F = F ∪ Hc∪ Hp. (15)Attorney Docket No.: 37684.6115WOM25-312P-WO1-bStep 7: Associate Low-confidence Children with Lonely Parents:
[0095] FIG. 11 illustrates adding low-confidence children class detections to high-confidence parent class detections. With reference to FIG. 11, the disclosed embodiments identify high-confidence lonely parents:for each d_i ∈ Hplonly, search for associated low-confidence children:
[0096] Add detections in XLcto the final detections:F = F ∪ XLc. (18)Step 8: Associate Low-confidence parents with Lonely Children:
[0097] FIG. 12 illustrates adding low-confidence children class detections to high-confidence parent class detections. With reference to FIG. 12, the disclosed embodiments identify high-confidence lonely childrenFor each d_j ∈ Hclonly, search for associated low-confidence parents:
[0098] Add detections in XLp_j to the final detections:F = F ∪ XLp_j. (21)Step 9: Resolve Newly Introduced Conflicts:
[0099] Reapply Steps 3 to 5 to eliminate any new conflicts caused by adding low-confidence parents.
[0100] Thus, the disclosed embodiments comprise a layer of logic on top of the YOLOv8 model that verifies the presence of at least one of the parts of an object to predict that object is in theAttorney Docket No.: 37684.6115WOM25-312P-WO1-bimage, with the exception being that although no parts were found (generally due to an adversarial attack), the object itself is predicted with high confidence. Normally, to predict that a destroyer is in an image, the algorithm verifies that at least one of the parts (e.g., the conning tower / bridge or the missile launcher or the helo deck or the gun) is visible at some level of confidence.
[0101] Thus, the disclosed embodiments provide a method for predicting an object exists in a digital image via an artificial intelligence (Al) model. With reference to FIG. 28, the process 2800 begins at step 2805 receiving user input defining a plurality of parts of the object, followed at step 2810 with receiving the digital image. Next, the process verifies at step 2815, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image. The process then predicts, at step 2820, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image.
[0102] According to an embodiment, the process may further identify at step 2825 a respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image. According to this embodiment, identifying the respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image comprises generating a bounding box superimposed on the digital image that identifies the respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image.
[0103] According to an embodiment, the process may further comprise generating a respective confidence score associated with the identified respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image, or the verified one or more of the user-defined plurality of parts of the object that exists in the digital image. According to this embodiment, predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the respective confidence score associated with the identified respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image, or the verified one or more of the user-defined plurality of parts of the object that exists in the digital image.
[0104] The disclosed embodiments may also generate a class associated with the object predicted to exist in the digital image.Attorney Docket No.: 37684.6115WOM25-312P-WO1-b
[0105] According to an embodiment, verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises proposing one or more regions in the digital image at which to search for an existence of one or more of the user-defined plurality of parts of the object in the digital image, and searching the proposed one or more regions in the digital image for the existence of one or more of the user-defined plurality of parts of the object in the digital image.
[0106] According to the disclosed embodiments, receiving user input defining the plurality of parts of the object comprises receiving user input from a first user defining a first plurality of parts of the object. According to this embodiment, verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, one or more of the first user-defined plurality of parts of the object exists in the digital image. According to this embodiment, predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the first user-defined plurality of parts of the object exists in the digital image. This embodiment further comprises the steps of receiving user input from a second user defining a second plurality of parts of the object that differ from the first plurality of parts of the object, verifying, via the Al model, one or more of the second user-defined plurality of parts of the object, and predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the one or more of the second user-defined plurality of parts of the object exists in the digital image.
[0107] According to this embodiment, receiving user input from the second user defining the second plurality of parts of the object that differ from the first plurality of parts of the object, comprises receiving user input from the second user defining a second plurality of parts of the object that differ in one of number, shape, form, or a combination thereof, from the first plurality of parts of the object.
[0108] According to the disclosed embodiments, verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, a portion of one or more of the user-defined plurality of parts of the object exists in the digital image. According to this embodiment, predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in theAttorney Docket No.: 37684.6115WOM25-312P-WO1-bdigital image responsive to the Al model verifying the portion of one or more of the user-defined plurality of parts of the object exists in the digital image.
[0109] The disclosed embodiment can also receive user input as to which of the one or more of the user-defined plurality of parts of the object the Al model shall verify, at a minimum, exists in the digital image. In such an embodiment, verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, the minimum one or more of the user-defined plurality of parts of the object exists in the digital image, and predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the minimum one or more of the user-defined plurality of parts of the object exists in the digital image.8. EXPERIMENTAL RESULTSA. MOTIVATION FOR TESTING WITH SATELLITE IMAGERY
[0110] The disclosed embodiments are focused on detecting ships and airplanes in satellite imagery mainly because there is interest in automated processing of satellite imagery on-board satellites. Under current technology, ground stations collect and process such imagery. But that process requires substantial expertise and is very labor intensive and costly. Often, due to lack of personnel, large volumes of imagery never get processed. Automated processing on-board satellites can solve some of these problems. In addition, with automated on-board processing, a satellite can quickly transmit only relevant information to the ground in symbolic form for timely decision-making and the imagery may be transmitted later for backup. But to automate this processing, highly reliable and explainable models are needed to reduce different types of risks including those arising from adversarial attacks.B. DETECTING SHIPS AND AIRPLANES FROM SATELLITE IMAGERY
[0111] Separate models were built to detect each object type (i.e., ships and airplanes) and its subclasses. For ships, the disclosed embodiments are focused on detecting larger ships like aircraft carriers, destroyers, and container and cargo ships. Parts for them that are quite large such as control towers, runways and containers are defined. In the airplane dataset, the planes were relatively small compared to the ships. The algorithm therefore was able to be tested on smaller objects with smaller parts. These models were subjected to adversarial attacks, both patch and camouflage type.Attorney Docket No.: 37684.6115WOM25-312P-WO1-bC. COULD NOT USE SOME STANDARD DATASETS FOR COMPARISON
[0112] Annotating parts of objects is quite labor intensive. Therefore, the XAI algorithm was not compared with object detection methods using standard datasets, such as Pascal VOC and COCO. Pascal VOC contains 20 object categories with 11,530 images and 27,450 object instances. COCO contains 80 object categories with 330K images and 1.5 million object instances.D. CHOOSING YOLO AS THE BASE MODEL; XAI r.s. non-XAI MODELS
[0113] Real-time, on-board image processing on satellites means not only having to deal with limited computational resources but also needing to consider processing speed. Of the various object detection algorithms, YOLO stands out as one of the fastest. The disclosed embodiments use the YOLOv8 XL version, which is a widely used model. Many others have used different versions of YOLO or new algorithms based on it in their studies of object detection from satellite imagery.
[0114] The disclosed embodiments demonstrate that beyond providing an explanation, the DARPA style XAI model provides robustness under adversarial attacks and improved accuracy.
[0115] The basic YOLOv8 model is herein called “non-XAI” because it is only trained with the main object classes (that is, ships and airplanes and their subclasses) with corresponding annotations; it is not provided with any information about parts of objects. The part-based YOLOv8 model according to the disclosed embodiments is referred to as “XAI” because it is trained with parts, defined as extra classes, along with the main object classes. The post-processing logic of Section 5 is also used in conjunction with the XAI YOLO model to verify and make the final predictions.E. CHOOSING YOLO AS THE BASE MODEL; XAI ra. non-XAI MODELS
[0116] The oriented bounding box (OBB) version of YOLOv8xl is used in the disclosed embodiments. The optimizer was stochastic gradient descent (SGD) and the default learning rates for SGD were used. The models were trained for 600 epochs with batch size 8 and image size 1280 x 1280 on Arizona State University’s SOL supercomputer. 4 vCPU cores and 16 GB of memory were used. The following thresholds were used: T0= 0.1;Ti = 0.25; 90= 0.7; 9 = 0.9.F. CHOOSING YOLO AS THE BASE MODEL; XAI ra. non-XAI MODELS
[0117] Even though satellite imagery data was obtained from standard sources, not all the imagery was used to build and test models. Subsets of images from these sources were used to evaluate the disclosed embodiments.Attorney Docket No.: 37684.6115WOM25-312P-WO1-b
[0118] Random sampling of the data subsets was used to generate the cross-validation splits (Monte Carlo cross-validation) and 4 different XAI and non-XAI models were trained and tested in both cases, that is, for ships and airplanes. The fraction of images in these train-test random partitions was on average between 75:25 and 80:20, and the test sets were inspected to minimize overlap and create balance. Part of the problem with any kind of sampling to create different train and test sets for cross-validation, even for k-fold cross-validation, is that the sampling is at the image level and may not produce a desired balance at the object level (e.g., destroyers, container ships, cargo ships) in the train-test splits. Hence, some manual image selection was completed to try to balance the splits at the object level classes.
[0119] Averages and standard deviations are reported from these cross-validation runs for the following measures: precision, recall, Fl -score, mAP50 and mAP50-95.G. SHIP DETECTION- SATELLITE IMAGERY DATA AND SOURCES
[0120] Ship images were used from the ShipRSImageNet collection. The ShipRSImageNet dataset has 532 images from xView, 1057 from HRSC2016, 1846 from FGSD, and 21 from the Airbus Ship Detection Challenge dataset. Additionally, 17 images were obtained by Chinese satellites.
[0121] ShipRSImageNet has three main classes - Merchant, Warship and Other Ship - and many subclasses of these main classes. These classes are redefined based on a collection of 323 images. In the “Other Ships (yachts, motorboats)” class in Table 1, presented in FIG. 13, yachts and small motorboats are mainly included, which are part of the “Merchant” class in the ShipRSImageNet dataset. No parts for “Other Ships (yachts, motorboats)” are defined because they are small. Only parts for three merchant ship classes are defined (container ships, cargo ships and a few other ship types are bundled together as subclass_l) and for three warship classes (destroyers, aircraft carriers and amphibious assault ships (LHA)). The non-XAI YOLO models are trained only with annotations for the main classes and the subclasses of ships, but not their parts.
[0122] Only 323 images were used from the ShipRSImageNet collection for training and testing both XAI and non-XAI YOLO models. Table 1 shows the counts for the main ship classes in these 323 images as per the definition of “Other Ships (yachts, motorboats).”H. SHIP DETECTION - PART DEFINITIONS
[0123] Table 2, presented in FIG. 14, shows the parts defined for the three merchant and three warship classes. The class “Warship destroyer” includes cruisers and frigates because they have similarAttorney Docket No.: 37684.6115WOM25-312P-WO1-bparts. Generally identifiable parts of these warships and merchant ships are used for the XAI model. They include: (1) the control tower (also called the bridge) labelled as the “warship_control,” “ac_control,” “lha_control,” and “merchant_control” depending on the vessel type, (2) the helicopter pad labelled as “helopad” and “lha_helopad” depending on the warship type, (3) the “vertrep” and “merchant_vertrep,” a place for supplying a ship using a helicopter (the process is called “vertical replenishment” where supplies are dropped from a helicopter), (4) the “gun” on a destroyer and cruiser where there are a variety of weapon systems, (5) the “launcher” on a destroyer and cruiser for launching a variety of missiles, (6) the “ac_runway” and “ac_launchpad” to distinguish the two runways on an aircraft carrier, (7) the “ac_plane” to identify individual planes on the deck of an aircraft carrier, (8) the “crane” usually seen on a merchant vessel, (9) the “container” for containers on a container ship (the disclosed embodiments may group a few containers together and annotate the container bundle), and (10) the “cargo” on a cargo ship.
[0124] Table 3, presented in FIG. 15, shows that the Warships and Merchant Ships categories contain some classes of ships for which parts are not defined in the disclosed embodiments. For example, parts for the Japanese warships (Warship_Japanese) or the submarines (Warship_submarine) are not identified. And there were other warships that are simply labeled as “Warship_unknown.” Similarly, parts for “merchant_barge,” “merchant_watercraft,” and “merchant_unknown” are not identified.
[0125] In the XAI YOLO models, the parts are additional classes of objects. Table 4, presented in FIG. 16, shows the total number of objects used to train and test XAI and non-XAI YOLO models. Note that XAI models have more objects compared to non-XAI models (4876 vs. 1969) because the various parts of different types of ships created additional classes. Table 10, presented in FIG. 22, shows the object count distribution for the ship detection dataset.I. SHIP DETECTION WITH PATCH TYPE ADVERSARIAL ATTACKS
[0126] Patch attacks of the type shown in FIG. 4, presented in FIG. 16, are used on parts of the ships (patches were placed only on ships that had parts defined), different shapes (round, square, rectangle etc.) and sizes are also used, and at random positions on those parts. The sizes of those patches were gradually increased so that they almost covered the object parts. Table 5, presented in FIG. 17, shows the results for XAI and non-XAI models with increasing size of the patches. Table 5 shows Average precision, recall, Fl -score, mAP50, and mAP50-95 over four random train-test splits for patch attack. For AVERAGES, “% Higher” column means that the XAI average is greater than the non-XAIAttorney Docket No.: 37684.6115WOM25-312P-WO1-baverage. For STANDARD DEVIATIONS, “% Higher” column means that the XAI standard deviation is lower than non-XAI. “Attack Strength” indicates the size of the patches. Attack Strength = 0 means there were no adversarial patch attacks. And Attack Strength = 2 has patches double the size of Attack Strength = 1.
[0127] Table 5, presented in FIG. 17, shows the averages and standard deviations of the various measures for XAI and non-XAI models for the six classes of ships for which parts are defined. Table 2, presented in FIG. 14, provides those part definitions.
[0128] Table 11, presented in FIG. 23, presents sample four-fold results from which were computed the averages and standard deviations of the various measures. Since there were different numbers of these six ship types in each of the four random test sets, the weighted averages for each of these measures (precision, recall, Fl -score, mAP50, and mAP50-95) for each test split was first calculated and then their averages and standard deviations over the four test splits were computed.
[0129] Table 5, presented in FIG. 17, shows the effectiveness of the patch attack as there is a significant decrease in all the average measures when Attack Strength is 2.
[0130] mAP50-95 is one of the main performance measures in object detection. Table 5, presented in FIG. 17, shows that XAI models according to the disclosed embodiments have on average about a 15% higher mAP50-95 value than the non-XAI models for all levels of patch attacks. And for “Attack Strength 2,” which represents a severe patch attack, XAI models have substantially higher percentage increases over non-XAI models for recall, Fl -score, mAP50 and mAP50-95. That indicates robustness of the XAI models compared to non-XAI models. Note that large patch attacks make large portions of the ship invisible.
[0131] FIGS 24A-24F provide some examples of patch attacks (see patches 2300) showing the advantage of XAI models according to the disclosed embodiment compared to non-XAI models. FIG.24A, in Case 1, shows a non-XAI model fails to detect the aircraft carrier under patch attack. FIG.24B, in Case 1, shows the XAI model finds the aircraft carrier and its parts (the control tower, the launchpad, the runway) even under patch attack. FIG. 24C, in Case 2, shows the non-XAI model misclassifies a part of the road as a merchant container ship. FIG. 24D, in Case 2, shows the XAI model does not classify the road as a merchant container ship. FIG. 24E, in Case 3, shows the non-XAI model misclassifies the dock area as a warship. And classifies the aircraft carrier, which is under patch attack,Attorney Docket No.: 37684.6115WOM25-312P-WO1-balso as a merchant ship. FIG. 24F, in Case 3, shows the XAI model does not classify the dock area as a warship. And correctly identifies the aircraft carrier and at least one of its parts - the control tower.J. SHIP DETECTION WITH CAMOUFLAGE ATTACKS
[0132] One form of camouflage is where one paints or covers military equipment to make them blend in with their surroundings so that they become invisible. An experiment was conducted in which the color of the ships was gradually changed to blend with the color of water surrounding them. FIGS.5, 6 and 7 show this gradual change of color for three ships in the image. The degree of color change is referred to as “Attack Strength.” Attack Strength = 0 means there was no color change. And Attack Strength = 100% means the ship blends with the surrounding water and becomes completely invisible. The experiment varied the Attack Strength from 25% to 75%, as shown in Table 6, presented in FIG.18.
[0133] Table 6 shows the average and standard deviations of the various measures over four folds for XAI and non-XAI models for the six classes of ships for which parts were defined. Averages and standard deviations were calculated the same way as for the patch attacks.
[0134] Table 6 also shows the effectiveness of the camouflage attack as there is significant decrease in all the average measures when the Attack Strength is 75%.
[0135] Table 6 shows that XAI models according to the disclosed embodiments have on average, about a 15% higher mAP50-95 value than the non-XAI models for all levels of camouflage attacks. And in the last column, with “Attack Strength: 75%”, which represents a severe camouflage attack, XAI models have substantially higher percentage increases over non-XAI models for all measures. That indicates robustness of the XAI models compared to non-XAI models.
[0136] FIGS. 25 A and 25B illustrate examples of camouflage attacks showing the advantage of XAI models (FIG. 25B) compared to non-XAI models (FIG. 25A). FIG. 25A shows a non-XAI model misclassifies a merchant ship as a warship. In particular, the non-XAI YOLO model misclassifies the “Merchant_subclass_l” ship as “Warship_ac_carrier,” a ship that is under camouflage attack. FIG. 25B shows the XAI YOLO model correctly classifies the “Merchant_subclass_l,” which is under camouflage attack, and identifies its two parts -marchant_helo and merchant_controlAttorney Docket No.: 37684.6115WOM25-312P-WO1-bK. AIRPLANE DETECTION - SATELLITE IMAGERY DATA AND SOURCES
[0137] The disclosed embodiments used aircraft images from the HRPlanesv2 collection and from the RarePlanes collection. HRPlanesv2 contains 2120 high resolution Google Earth images that have 14,335 aircraft. They include images from different satellites of different airports around the world and under clear weather conditions. It includes a mix of civil, military and joint aircraft. HRPlanesv2 also presents results of training a Y0L0v8m model. The RarePlanes dataset is also high resolution and contains 253 real and 50,000 synthetically generated satellite imagery. The 253 real images are from Maxar WorldView-3 satellites and have 14,700 hand-annotated aircraft. The 50,000 synthetic images have 630,000 aircraft annotations. The following sites contain these datasets:HRPlanes - zenodo. org / records / 7331974RarePlanes- www.cosmiqworks.or.g / rareplanes /
[0138] Experiments with the disclosed embodiments collected only 159 images from these two sources to train and test both XAI and non-XAI YOLO models. The experiments trained the models to identify three types of airplanes - fighter, transport, and civil planes. Table 7, presented in FIG. 19, shows the counts for each type of plane. The disclosed embodiments used the fuselage, the wings and the tail as parts of these planes. The disclosed embodiments distinguished the parts for each type of plane by simply tagging them accordingly. Thus, civil-planes have civil-fuselage, civil-tail, and civil-wings. Table 7 also shows the counts of these part objects. As usual, the disclosed embodiments trained the non-XAI YOLO models with annotations for the planes only, not the parts. Note that XAI models have more objects compared to non-XAI models (7163 vs. 1440) because the various parts of different types of planes created additional classes and objects.L. AIRPLANE DETECTION WITH PATCH TYPE ADVERSARIAL ATTACKS
[0139] As in the case of ships, the disclosed embodiments used patch attacks of the type shown in FIG. 4 on parts of the airplanes, used different shapes (round, square, rectangle etc.) and sizes, and at random positions on those parts. The disclosed embodiments gradually increased the sizes of those patches so that they almost covered the object parts. Table 8, presented in FIG. 20, shows the results for XAI and non-XAI models with increasing size of the patches.
[0140] Table 8 shows the averages and standard deviations of the various measures for XAI and non-XAI models for the three types of planes. Since there were different numbers of these airplane types in each of the 4 random test sets, the weighted averages for each of these measures (precision,Attorney Docket No.: 37684.6115WOM25-312P-WO1-brecall, Fl -score, mAP50, and mAP50-95) for each test split was first calculated and then their averages and standard deviations over the 4 test splits were computed.
[0141] Table 8 shows the effectiveness of the patch attack as there is a significant decrease in all the average measures when Attack Strength is 2.
[0142] Table 8 shows that our XAI models on average have higher values than non-XAI models for all levels of patch attacks. And for “Attack Strength 2,” which represents a severe patch attack, XAI models have substantially higher percentage increases over non-XAI models for precision, recall, Fl -score, and mAP50. That indicates robustness of the XAI models compared to non-XAI models. Note that large patch attacks make large portions of the airplane invisible.
[0143] FIGS. 26A-26D provide some examples of patch attacks showing the advantage of XAI models compared to non-XAI models. FIG. 26A shows, in Case 1, the non-XAI YOLO model properly identified at 2605 some airplanes in this image even though they had large patch attacks 2600. However, it missed two airplanes. The circles 2610 mark the two missed planes which have large patch attacks 2600. FIG. 26B shows, in Case 1, The XAI YOLO model properly identified all airplanes in this image even under patch attack as indicated by the bounding boxes that surround each airplane. The fuselage is surrounded by a bounding box 2615, the wings are surrounded by bounding boxes 2620 in two parts, and the tail is surrounded by bounding boxes 2625. Note that planes with large patch attacks were correctly identified. FIG. 26C shows, in Case 2, the non-XAI YOLO model correctly identifies certain airplanes in this image which are under patch attack, as indicated by the rectangular bounding boxes. But the model misses many others - circles mark the missed planes which have large patch attacks. FIG. 26D shows, in Case 2, the XAI YOLO model correctly identifies all the planes in this image despite the patch attacks. Note that in all cases the tail was correctly identified and surrounded by a bounding boxed. And in some cases, the wings were identified and surrounded by a bounding box. However, all planes were correctly identified despite large patch attacks.M. AIRPLANE DETECTION WITH CAMOUFLAGE ATTACKS
[0144] Table 9, presented in FIG. 21, shows the average and standard deviations of the various measures over four folds for XAI and non-XAI models for the three types of planes. The averages and standard deviations were calculated the same way as for the patch attacks. Table 9 also shows the effectiveness of the camouflage attack as there is a significant decrease in all the average measures when the Attack Strength is 75%. And for “Attack Strength: 75%, ” which represents a severeAttorney Docket No.: 37684.6115WOM25-312P-WO1-bcamouflage attack, XAI models have substantially higher percentage increases over non-XAI models for all measures. That indicates robustness of the XAI models compared to non-XAI models.
[0145] FIGS. 27 A - 27D provide some examples of camouflage attacks showing the advantage of XAI models compared to non-XAI models. FIG. 27 A, in Case 1, shows the non-XAI YOLO model finds only one of the four planes, all of which are under camouflage attack at strength = 0.75. FIG.27B, in Case 1, shows the XAI YOLO model finds all four planes that are under camouflage attack at strength = 0.75, including some of their parts. FIG. 27C, in Case 2, shows the non-XAI YOLO model finds only six of the eight planes, all of which are under camouflage attack at strength = 0.75. FIG 27D, in Case 2, shows the XAI YOLO model finds all eight planes that are under camouflage attack at strength = 0.75, including some of their parts.9. CONCLUSIONS
[0146] In standard deep learning approaches to computer vision and object detection, one builds models first and thinks about explainability later. The disclosed embodiments reverse that process. According to the disclosed embodiments, one specifies the form of explanation first and then builds models according to those specifications.
[0147] For example, in working with medical images, physicians are asked what specific biomarkers they look for when identifying certain diseases. In the case of Alzheimer’s, they typically examine hippocampal atrophy, cortical thinning, and ventricular enlargement in MRIs. Under the DARPA framework, those biomarkers correspond to “parts” of the disease profile, and XAI object detection models are built to identify them. This ensures inherent explainability of the models according to the disclosed embodiments. The process is analogous to an architect meeting with clients to determine their needs before designing and constructing a house. The XAI modeling process according to the disclosed embodiments is fundamentally user centric.
[0148] The EU Al Act aims to standardize forms of explanation to enhance transparency, ensure compliance, and build public trust in Al technologies. The DARPA framework aligns with these goals by offering structured, interpretable explanations that facilitate regulatory adherence and broader Al adoption.
[0149] Unlike traditional visual explanation methods like saliency maps, which often require human interpretation and manual verification, the disclosed embodiments generate explanations continuously, without human intervention. These explanations can be stored for future analysis,Attorney Docket No.: 37684.6115WOM25-312P-WO1-bmaking them particularly valuable for automated, real-time processing in critical applications such as satellite and drone imagery, CCTV surveillance, and robotics. Additionally, this structured form of explainability integrates seamlessly with neuro-symbolic Al, allowing for the incorporation of contextual information in decision-making.
[0150] Post-hoc explainability methods often force a trade-off between accuracy and interpretability by constraining models to use human-comprehensible variables. Research shows that when explainability is built into the model from the start, human-understandable features enhance accuracy and performance rather than compromise them.
[0151] As mentioned before, because annotating parts of objects is labor intensive, it was not possible to compare the XAI model according to the disclosed embodiments with the non-XAI YOLO model using some of the standard datasets, such as Pascal VOC and COCO, that have thousands, if not millions, of object instances. However, requiring part annotations for thousands of object instances is not a severely limiting factor for the XAI model according to the disclosed embodiments as there is a thriving industry nowadays for doing exactly that. So, for critical applications, resource availability for performing annotations should not be a hindrance.
Claims
Attorney Docket No.: 37684.6115WOM25-312P-W01-bCLAIMSWhat is claimed is:
1. A method for predicting an object exists in a digital image via an artificial intelligence (Al) model, comprising:receiving user input defining a plurality of parts of the object;receiving the digital image;verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image;predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image.
2. The method of claim 1, further comprising identifying a respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image.
3. The method of claim 2, wherein identifying the respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image, comprises generating a bounding box superimposed on the digital image that identifies the respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image.
4. The method of claim 2, further comprising generating a respective confidence score associated with the identified respective location of the verified one or more of the user- defined plurality of parts of the object that exists in the digital image, or the verified one or more of the user-defined plurality of parts of the object that exists in the digital image.
5. The method of claim 4, wherein predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the AlAttorney Docket No.: 37684.6115WOM25-312P-W01-bmodel, the object exists in the digital image responsive to the respective confidence score associated with the identified respective location of the verified one or more of the user- defined plurality of parts of the object that exists in the digital image, or the verified one or more of the user-defined plurality of parts of the object that exists in the digital image.
6. The method of claim 1, further comprising generating a class associated with the object predicted to exist in the digital image.
7. The method of claim 1, wherein verifying, via the Al model, one or more of the user- defined plurality of parts of the object exists in the digital image, comprises: proposing one or more regions in the digital image at which to search for an existence of one or more of the user-defined plurality of parts of the object in the digital image; and searching the proposed one or more regions in the digital image for the existence of one or more of the user-defined plurality of parts of the object in the digital image.
8. The method of claim 1,wherein receiving user input defining the plurality of parts of the object comprises receiving user input from a first user defining a first plurality of parts of the object;wherein verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, one or more of the first user-defined plurality of parts of the object exists in the digital image; wherein predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the first user-defined plurality of parts of the object exists in the digital image;the method further comprising:receiving user input from a second user defining a second plurality of parts of the object that differ from the first plurality of parts of the object;Attorney Docket No.: 37684.6115WOM25-312P-W01-bverifying, via the Al model, one or more of the second user-defined plurality of parts of the object; andpredicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the one or more of the second user-defined plurality of parts of the object exists in the digital image.
9. The method of claim 8, wherein receiving user input from the second user defining the second plurality of parts of the object that differ from the first plurality of parts of the object, comprises receiving user input from the second user defining a second plurality of parts of the object that differ in one of number, shape, form, or a combination thereof, from the first plurality of parts of the object.
10. The method of claim 1, wherein verifying, via the Al model, one or more of the user- defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, a portion of one or more of the user-defined plurality of parts of the object exists in the digital image; andwherein predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the portion of one or more of the user- defined plurality of parts of the object exists in the digital image.
11. The method of claim 1, further comprising receiving user input as to which of the one or more of the user-defined plurality of parts of the object the Al model shall verify, at a minimum, exists in the digital image; andwherein verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, the minimum one or more of the user-defined plurality of parts of the object exists in the digital image; andwherein predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists inAttorney Docket No.: 37684.6115WOM25-312P-W01-bthe digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the minimum one or more of the user-defined plurality of parts of the object exists in the digital image.
12. A system comprising:a memory to store instructions;a processor to execute the instructions stored in the memory to cause the system to perform operations including:receiving user input defining a plurality of parts of the object;receiving the digital image;verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image;predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image.
13. The system of claim 12, further comprising:identifying a respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image, wherein identifying the respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image, comprises generating a bounding box superimposed on the digital image that identifies the respective location of the verified one or more of the user- defined plurality of parts of the object that exists in the digital image.
14. The system of claim 12,wherein receiving user input defining the plurality of parts of the object comprises receiving user input from a first user defining a first plurality of parts of the object;wherein verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, one or more of the first user-defined plurality of parts of the object exists in the digital image;Attorney Docket No.: 37684.6115WOM25-312P-W01-bwherein predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the first user-defined plurality of parts of the object exists in the digital image;the method further comprising:receiving user input from a second user defining a second plurality of parts of the object that differ from the first plurality of parts of the object;verifying, via the Al model, one or more of the second user-defined plurality of parts of the object; andpredicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the one or more of the second user-defined plurality of parts of the object exists in the digital image.
15. The system of claim 12, wherein verifying, via the Al model, one or more of the user- defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, a portion of one or more of the user-defined plurality of parts of the object exists in the digital image; andwherein predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the portion of one or more of the user- defined plurality of parts of the object exists in the digital image.
16. The system of claim 12, further comprising receiving user input as to which of the one or more of the user-defined plurality of parts of the object the Al model shall verify, at a minimum, exists in the digital image; andwherein verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, the minimumAttorney Docket No.: 37684.6115WOM25-312P-W01-bone or more of the user-defined plurality of parts of the object exists in the digital image; andwherein predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the minimum one or more of the user-defined plurality of parts of the object exists in the digital image.
17. A computer readable storage media having instructions stored thereupon that, when executed by a process of a system, cause the system to perform operations including: receiving user input defining a plurality of parts of the object;receiving the digital image;verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image;predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image.
18. The computer readable storage media of claim 17, further comprising:identifying a respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image, wherein identifying the respective location of the verified one or more of the user-defined plurality of parts of the object that exists in the digital image, comprises generating a bounding box superimposed on the digital image that identifies the respective location of the verified one or more of the user- defined plurality of parts of the object that exists in the digital image.
19. The computer readable storage media of claim 17,wherein receiving user input defining the plurality of parts of the object comprises receiving user input from a first user defining a first plurality of parts of the object;Attorney Docket No.: 37684.6115WOM25-312P-W01-bwherein verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, one or more of the first user-defined plurality of parts of the object exists in the digital image; wherein predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the first user-defined plurality of parts of the object exists in the digital image;the method further comprising:receiving user input from a second user defining a second plurality of parts of the object that differ from the first plurality of parts of the object;verifying, via the Al model, one or more of the second user-defined plurality of parts of the object; andpredicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the one or more of the second user-defined plurality of parts of the object exists in the digital image.
20. The computer readable storage media of claim 17, wherein verifying, via the Al model, one or more of the user-defined plurality of parts of the object exists in the digital image, comprises verifying, via the Al model, a portion of one or more of the user-defined plurality of parts of the object exists in the digital image; andwherein predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying one or more of the user-defined plurality of parts of the object exists in the digital image, comprises predicting, via the Al model, the object exists in the digital image responsive to the Al model verifying the portion of one or more of the user- defined plurality of parts of the object exists in the digital image.
