External pod for multi-modal sensor fusion in aircraft

Abstract

A method and system employing an external pod for an aircraft having an integrated radar system including: an electro-optical (E / O) sensor suite configured to collect E / O data; a communication interface configured to receive radar-derived data from the aircraft's integrated radar system; and a processing system including at least one processor, the processing system including: an E / O data processing module configured to process the E / O data; and a data fusion module configured to fuse the processed E / O data with the radar-derived data to generate a fused multi-modal information output.

Description

[0001] External Pod for Multi-Modal Sensor Fusion in Aircraft

[0002] FIELD AND BACKGROUND OF THE INVENTION

[0003] This invention relates to the field of aircraft sensor systems, and more particularly to external pods for aircraft that integrate and fuse data from multiple sensor modalities.

[0004] Modern aircraft, particularly those used in military or surveillance applications, often employ a variety of sensors to gather information about their surroundings. These sensors may include radar systems, electro-optical (E / O) sensors operating in various wavelengths (e.g., visible, infrared), and other types of sensors.

[0005] Optimal utilization of the data from these diverse sensors can be achieved by integrating and fusing the data from different modalities. This sensor fusion process can enhance situational awareness, improve target identification, and enable more effective operation of the aircraft, particularly in challenging environments. The latest generation of combat aircraft typically feature integrated E / O sensor suites alongside their radar systems, and these aircraft possess native systems for fusing data from these different modalities. However, many older aircraft lack such integrated E / O capabilities. While advanced E / O sensor suites can be added to these aircraft through external pods, they still lack the advantages of multi-modal data fusion.

[0006] SUMMARY OF THE INVENTION

[0007] The present invention is an external pod for aircraft that integrates and fuses data from multiple sensor modalities.

[0008] According to the teachings of an embodiment of the present invention there is provided, an external pod for an aircraft having an integrated radar system, the pod comprising: (a) an electro- optic al (E / O) sensor suite configured to collect E / O data; (b) a communication interface configured to receive radar-derived data from the aircraft’ s integrated radar system; and (c) a processing system including at least one processor, the processing system including: (i) an E / O data processing module configured to process the E / O data; and (ii) a data fusion module configured to fuse the processed E / O data with the radar-derived data to generate a fused multi-modal information output.According to a further feature of an embodiment of the present invention, the E / O sensor suite comprises at least two sensors operating in different E / O wavelength bands.

[0009] According to a further feature of an embodiment of the present invention, the communication interface is configured to receive the radar data via a data bus of the aircraft.

[0010] According to a further feature of an embodiment of the present invention, the communication interface is further configured to receive data from at least one external source via a wireless communication subsystem, and wherein the processing system is further configured to fuse the processed E / O data and the radar data with the data from the at least one external source.

[0011] According to a further feature of an embodiment of the present invention, the at least one external source is selected from the group consisting of: another aircraft, a ground station, and a SIGINT pod.

[0012] According to a further feature of an embodiment of the present invention, the fused multi-modal information output comprises at least one of: target identification data, target tracking data, and target range information.

[0013] According to a further feature of an embodiment of the present invention, there is also provided a user interface module located within the aircraft configured to display the fused multi-modal information output to a user of the aircraft.

[0014] According to a further feature of an embodiment of the present invention, the user interface module is further configured to allow the user of the aircraft to control at least one function of the pod.

[0015] According to a further feature of an embodiment of the present invention, the communication interface is further configured to provide the fused multi-modal information output to at least one of: a display of the aircraft and a weapon system of the aircraft.

[0016] According to a further feature of an embodiment of the present invention, the processing system is further configured to generate guidance information for at least one guided weapon of the aircraft based on the fused multi-modal information output.

[0017] There is also provided according to an embodiment of the present invention, an external pod for an aircraft, the pod comprising: (a) an electro-optical (E / O) sensor suite configured to collect E / O data; (b) a communication interface configured to receive data from at least oneexternal source via a wireless communication subsystem; and (c) a processing system including at least one processor, the processing system including: (i) an E / O data processing module configured to process the E / O data; and (ii) a data fusion module configured to fuse the processed E / O data with the data from the at least one external source to generate a fused information output.

[0018] There is also provided according to the teachings of an embodiment of the present invention, a method for providing to a user of an aircraft enhanced data from an electro- optic al (E / O) targeting pod carried by the aircraft, the method comprising the steps of: (a) employing an E / O sensor within the targeting pod to sample images; (b) processing the images to derive a first set of target-related data; (c) receiving via a communication interface of the pod a data stream originating outside the targeting pod containing a second set of target-related data; (d) performing data fusion on the first and second sets of target-related data to generate enhanced target-related data; and (e) making the enhanced target-related data available to the user of the aircraft.

[0019] According to a further feature of an embodiment of the present invention, the data stream originating outside the targeting pod contains radar-derived data derived from a radar system of the aircraft.

[0020] According to a further feature of an embodiment of the present invention, the communication interface includes a wireless communication subsystem for receiving the data stream originating outside the targeting pod from a source remote from the aircraft.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The invention is herein described, by way of example only, with reference to the accompanying drawings, wherein:

[0023] FIG. 1 is a schematic diagram of a prior art integrated multi-modal sensor system in an aircraft;

[0024] FIG. 2 is a schematic diagram of a prior art system in which an E / O pod is attached to an aircraft; and

[0025] FIG. 3 is a schematic diagram of an aircraft equipped with an external pod according to an embodiment of the invention.DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The present invention is an external pod for aircraft that integrates and fuses data from multiple sensor modalities.

[0027] By way of introduction, modern combat aircraft often utilize a variety of sensors to gather information about their surroundings and to detect and identify targets. Such sensors may include radar systems and electro-optical (E / O) sensors operating in various wavelengths (e.g., visible, infrared). The latest generation of combat aircraft typically feature integrated E / O sensor suites alongside their radar systems, and possess native systems for fusing data from these different modalities, as shown in FIG. 1. The fusion of data from multiple sensors (multi-modal data fusion) enhances situational awareness, improves target identification, and enables more effective operation of the aircraft, particularly in challenging environments.

[0028] FIG. 1 shows schematically a fifth-generation fighter jet 100 with an integrated multimodal sensor system. The aircraft 100 includes a radar system 102, a radar processor 104, an E / O sensor suite 106, an E / O processor 108, a data fusion processor 110, and a user interface 112. Data from the radar system 102 is processed by the radar processor 104, and data from the E / O sensor suite 106 is processed by the E / O processor 108. The data fusion processor 110 fuses the processed data from the radar processor 104 and the E / O processor 108 and provides an output to the user interface 112. Use of data fusion enhances the functionality which could be achieved by each modality alone, offering enhanced and more robust target detection, ranging, identification and tracking, and providing all-weather capabilities.

[0029] Older aircraft typically do not have such integrated E / O capabilities, or at least lack E / O capabilities of modern high-quality optical systems. In such aircraft, advanced E / O sensor suites can be added through external pods, as shown in FIG. 2.

[0030] FIG. 2 shows a fighter jet 200 with an E / O pod 202 attached externally thereto. The aircraft 200 includes a radar system 204, a radar processor 206, a radar user interface 208, a data bus 210, and an E / O user interface 212 which is implemented as an upgraded or supplemental interface to accompany pod 202. The E / O pod 202 includes an E / O sensor suite 214 and an E / O processor 216. The E / O processor 216 typically receives, via the data bus 210, low-bandwidth data such as location and airspeed data (as illustrated by the narrow bandwidth arrow 218). The E / O processor 216 processes data from the E / O sensor suite 214 and providesoutputs derived from the sensors to aircraft 200 via data bus 210 (as illustrated by the wide bandwidth arrow 220). The information from pod 202 and control of the pod functions is achieved by the associated E / O user interface 212.

[0031] While such pods provide enhanced E / O functionality, they fail to provide the advantages of multi-modal data fusion. The present invention addresses this limitation by providing an external pod that integrates and fuses data from multiple sensor modalities, including data from the aircraft's radar system and E / O sensors in the pod, as will now be detailed with reference to FIG. 3.

[0032] FIG. 3 shows an aircraft 10 equipped with an external pod 12 according to an embodiment of the invention. The aircraft 10 includes a radar system 14, a radar processor 16, a radar user interface 18, a data bus 20, a dedicated user interface 26, and a dedicated weapon controller 28. The aircraft 10 also includes a weapon 24. The E / O pod 12 includes an E / O sensor suite 30, a processing system 34, a communication interface 46, and a wireless communication module 48. The processing system 34 includes an E / O data processing module 36 and a data fusion module 38. The E / O data processing module 36 processes data from the E / O sensor suite 30. The data fusion module 38 fuses the processed E / O data with radar-derived data received from the aircraft's radar system 14 via communication interface 46 and data bus 20. The data fusion module 38 also receives low-bandwidth data from the data bus 20. The communication interface 46 may also receive data from an external ground station 50 or other external sources discussed further below via the wireless communication module 48.

[0033] It will immediately be appreciated that embodiments of the present invention provide several advantages over conventional external pod aircraft sensor systems. Specifically, by making radar-derived data from the aircraft radar system available to pod 12 via data bus 20 and by providing a data fusion module 38 as part of the pod processing system 34, the system provides a user (pilot or other crew member) the many advantages of multi-modal data fusion for an aircraft which lacks such native functionality. Furthermore, using an edge computing implementation with sufficient processing power in the pod processing system 34, this functionality can be provided without overburdening onboard aircraft processing systems, and even in cases where the onboard systems lack sufficient processing power to perform the required data fusion processing. The invention thus makes available cutting-edge multi-modaldata fusion comparable to that previously available only in the most modern generation of fighter aircraft, integrating and fusing data from multiple sensor modalities including both E / O and radar data.

[0034] The fused multi-modal information output can be used for various purposes, including but not limited to target tracking, target identification, and weapon guidance. The dedicated user interface 26 provides the user with a clear and comprehensive display of the fused sensor information, enabling more effective decision-making and operational performance. The dedicated weapon controller 28 allows the user to control the aircraft's weapons based on the fused sensor information and / or to hand-off enhanced target tracking data directly to a weapon system, improving the accuracy and effectiveness of weapon deployment.

[0035] At this point, it will be helpful to define certain terminology as used herein in the description and claims.

[0036] • Multi-modal: Refers to the use of data from different types of sensors, and in particular, which generate fundamentally different types of output data, such as radar, which generates a range map of radar reflectance values, and electro-optical sensors, which generate a two-dimensional image of textures corresponding to reflected light or emitted thermal radiation.

[0037] • E / O sensor suite: A collection of electro-optical sensors operating in different wavelength bands, such as visible light, infrared, and thermal imaging.

[0038] • Data: Refers herein primarily to digital or analogue signals that are received from a wide variety of sensors that sense radiation anywhere within the electromagnetic spectrum, including but not limited to optical sensors of visible light, near-infrared or thermal wavelengths, microwaves and radio waves.

[0039] • Radar-derived data: Information obtained from the aircraft's radar system, which may be processed or raw data.

[0040] • Fused multi-modal information output: The result of combining and processing data from multiple sensor modalities, including E / O and radar-derived data, to derive provide enhanced information beyond what could be derived from any one modality alone.• Processing: Encompasses various data manipulation techniques applied to the sensor data, including filtering, noise reduction, feature extraction, analysis, and any other suitable data processing operations, optionally enhanced or replaced by use of artificial intelligence processing for raw data processing and / or for fusion processing of data from multiple sources.

[0041] • Target: Any object or entity of interest that the system is intended to detect, track, or identify, including but not limited to aircraft, vehicles, structures, and individuals.

[0042] • Guidance information: Data generated by the processing system to guide weapons, including parameters such as target location, trajectory prediction, and timing information.

[0043] • Low-bandwidth data: Relatively low-rate data transmitted between the aircraft and the pod, including aircraft location, altitude, speed, heading. The low-bandwidth data may also include control data input by the user or otherwise generated by the aircraft systems to direct the pod to perform various operations, such as directing the E / O sensors to investigate an object detected by the radar system, to track a target or to execute a scanning pattern to derive a large area image or to search for a target.

[0044] • High-bandwidth data: Relatively high-rate data transmitted between the aircraft and the pod, including either raw data from a sensor system or rapidly changing data derived from processing such a data stream and / or combining information from such data streams.

[0045] • The term “remote” is used herein in the description and claims to refer to any source or object which does not move together with (i.e., in the same frame of reference as) an aircraft.

[0046] Turning now to details of certain preferred implementations of the present invention in more detail, E / O sensor suite 30 in pod 12 typically includes at least two sensors operating in different E / O wavelength bands. These sensors may include, for example, a visible light camera, an infrared camera, and / or a thermal imaging camera. The different wavelength bands provide complementary information, enabling the system to detect and identify targets under various conditions, including day, night, and adverse weather.Communication interface 46 receives radar-derived data from the aircraft's integrated radar system 14, typically after processing by radar processing system 16 to derive candidate objects of interest, and low-bandwidth data from the aircraft 10 via data bus 20. Communication interface 46 preferably also includes a wireless communication module 48 to receive data from one or more external sources, such as another aircraft, another pod, a UAV or other airborne platform, a satellite, a ground station 50, each of which may include additional sensor systems and / or may relay information from other sources such as a SIGINT pod. This allows the system to incorporate information from a wider range of sources, further enhancing situational awareness and target identification capabilities. Wireless communications module 48 may also be used to share processed data from data fusion module 38 with other pods or other remote platforms.

[0047] Although described thus far in an implementation that combines E / O-derived data from the pod with radar-derived data from the aircraft through multimodal data fusion, alternative implementations of the present invention employ the same principles for data fusion between E / O-derived data from the pod and some other source of data which is external to the pod, typically input via wireless communication module 48, even in the absence of radar input from the aircraft. Potential sources of data for such an implementation are similar to those mentioned above, but where the externally supplied data becomes the primary data to be combined with E / O-derived data through fusion processing. Thus, possible external sources include, but are not limited to, E / O sensors or radar sensors mounted on another aircraft, another pod, a UAV or other airborne platform, a satellite, or a ground station 50, or target information derived from a SIGINT system. In the case of E / O sensors mounted on another platform, the input data may not necessarily be “multimodal”, but a data fusion process is required due to the differing viewpoints, motion, optical properties and / or sampling wavelengths of the data sources.

[0048] Processing system 34 in pod 12 includes at least one processor and is responsible for processing the E / O data and fusing it with the radar-derived data. For this purpose, processing system 34 includes an E / O data processing module 36 and a data fusion module 38. The processing system is preferably configured to perform edge computing, enabling real-time processing of the sensor data within the pod and real-time fusion processing. The E / O dataprocessing module 36 processes the raw data from the E / O sensor suite 30. Data fusion module 38 combines the processed E / O data with the radar-derived data and the low-bandwidth data, as well as any other available local or remote data sources, to generate a fused multi-modal information output. The processing system 34 may also be configured to generate guidance information for guided weapons 24 of aircraft 10 based on the fused multi-modal information output.

[0049] It will be noted that the multi-modal data fusion processing, per se, is known in the art, as evidenced by the conventional integrated systems of fifth generation fighter aircraft. The data fusion is typically based on artificial intelligence (Al) processing through trained artificial neural networks that learn to correlate targets detected via multiple modalities and to combine the information available from each modality to enhance target acquisition, target identification, target ranging, target tracking and all-weather robustness of any and all of the above. Although Al processing is believed to be particularly suited to these tasks, alternative implementations also fall within the scope of the present invention. For example, deterministic algorithms may be employed to correlate positions of signals from different modalities and add range information to optically-detected targets to allow precise location of targets in three dimensions. All such implementations are generally known, and are within the capabilities of a person having ordinary skill in the art. For conciseness of presentation, further details of the fusion processing will not be described herein.

[0050] To allow user interaction with pod 12, a complementary user interface module 26 is located within aircraft 10. User interface module 26 is configured to display the fused multimodal information output to a user of the aircraft 10. User interface module 26 preferably also allows the user of aircraft 10 to control at least one function of pod 12 by inputs to the user interface module which are relayed to the pod via communication interface 46. These user controls may include controls to direct the pod to perform various operations, such as directing the E / O sensors to investigate an object detected by the radar system, to track a target or to execute a scanning pattern to derive a large area image or to search for a target.

[0051] The invention is not limited to the specific embodiment described above. Various modifications and variations are possible within the scope of the invention. For example, the specific types of E / O sensors included in the E / O sensor suite 30 may vary depending on thespecific application and requirements. The communication interface 46 may be configured to communicate with different types of external sources, and the processing system 34 may be implemented using various combinations of hardware and software configurations.

[0052] It will be appreciated that the above descriptions are intended only to serve as examples, and that many other embodiments are possible within the scope of the present invention as defined in the appended claims.

Claims

WHAT IS CLAIMED IS:

1. An external pod for an aircraft having an integrated radar system, the pod comprising:(a) an electro-optical (E / O) sensor suite configured to collect E / O data;(b) a communication interface configured to receive radar-derived data from the aircraft’ s integrated radar system; and(c) a processing system including at least one processor, the processing system including:(i) an E / O data processing module configured to process the E / O data; and (ii) a data fusion module configured to fuse the processed E / O data with the radar-derived data to generate a fused multi-modal information output.

2. The pod of claim 1, wherein the E / O sensor suite comprises at least two sensors operating in different E / O wavelength bands.

3. The pod of claim 1, wherein the communication interface is configured to receive the radar data via a data bus of the aircraft.

4. The pod of Claim 1, wherein the communication interface is further configured to receive data from at least one external source via a wireless communication subsystem, and wherein the processing system is further configured to fuse the processed E / O data and the radar data with the data from the at least one external source.

5. The pod of claim 4, wherein the at least one external source is selected from the group consisting of: another aircraft, a ground station, and a SIGINT pod.

6. The pod of claim 1, wherein the fused multi-modal information output comprises at least one of: target identification data, target tracking data, and target range information.

7. The pod of claim 1, further comprising a user interface module located within the aircraft configured to display the fused multi-modal information output to a user of the aircraft.

8. The pod of claim 7, wherein the user interface module is further configured to allow the user of the aircraft to control at least one function of the pod.

9. The pod of claim 1, wherein the communication interface is further configured to provide the fused multi-modal information output to at least one of: a display of the aircraft and a weapon system of the aircraft.

10. The pod of claim 1, wherein the processing system is further configured to generate guidance information for at least one guided weapon of the aircraft based on the fused multimodal information output.

11. An external pod for an aircraft, the pod comprising:(a) an electro-optical (E / O) sensor suite configured to collect E / O data;(b) a communication interface configured to receive data from at least one external source via a wireless communication subsystem; and(c) a processing system including at least one processor, the processing system including:(i) an E / O data processing module configured to process the E / O data; and (ii) a data fusion module configured to fuse the processed E / O data with the data from the at least one external source to generate a fused information output.

12. A method for providing to a user of an aircraft enhanced data from an electro-optical (E / O) targeting pod carried by the aircraft, the method comprising the steps of:(a) employing an E / O sensor within the targeting pod to sample images;(b) processing the images to derive a first set of target-related data;(c) receiving via a communication interface of the pod a data stream originating outside the targeting pod containing a second set of target-related data;(d) performing data fusion on the first and second sets of target-related data to generate enhanced target-related data; and(e) making the enhanced target-related data available to the user of the aircraft.

13. The method of claim 12, wherein the data stream originating outside the targeting pod contains radar-derived data derived from a radar system of the aircraft.

14. The method of claim 12, wherein the communication interface includes a wireless communication subsystem for receiving the data stream originating outside the targeting pod from a source remote from the aircraft.

Images