Multi-Sensor Image Fusion for Zero Visibility Flight
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
During flight operations in near-zero to zero visibility conditions due to obscurants like rain, snow, or dust, pilots face limited vision, and existing aircraft systems, such as radar, are insufficient to enable safe taxiing, taking off, or landing.
Innovation Solution
A system that generates a synthetic 3D representation of the environment by combining infrared data from an IR camera, laser point cloud data from LIDAR, and navigation information, transforming the data into a geo-referenced coordinate space, and overlaying it with terrain imagery to create a real-time synthetic image for display on multifunction or helmet-mounted displays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If radar is used to provide additional information about surroundings, then the pilot receives some information about the environment, but the information is insufficient to enable safe taxiing, take-off and landing in conditions with significant obscurants
Solution Approach 1:
The patent combines multiple sensor types (LIDAR, infrared cameras, radar, visible light cameras) to create a fused multi-dimensional representation of the environment. This merging of different sensing modalities provides comprehensive spatial, thermal, and reflective information that overcomes the limitations of radar alone, enabling safe operation in obscured conditions by presenting a complete picture of the surroundings.
Solution Approach 2:
The patent transforms 2D sensor data into a 3D geo-referenced coordinate space, adding spatial dimensionality to the environmental representation. This dimensional transformation creates a synthetic reality that provides depth, distance, and spatial relationships, enabling pilots to perceive the environment in three dimensions even when visual cues are obscured by weather conditions.
2Loss of information
If multiple sensors are combined to generate synthetic 3D representation, then situational awareness is enhanced, but the system complexity increases
Solution Approach 1:
The patent segments the complex sensing and processing system into distinct functional modules: LIDAR data acquisition, infrared imaging, radar sensing, navigation database integration, and synthetic image generation. Each module processes specific types of data independently before fusion, making the overall complex system manageable through modular architecture and specialized processing pipelines.
Solution Approach 2:
The patent introduces a navigation database as an intermediary that stores pre-acquired multi-dimensional environmental information. This intermediary component bridges the gap between raw sensor data and the final synthetic representation, providing a reference framework for geo-referencing and integrating real-time sensor data with stored environmental models, thereby simplifying the fusion process.
3Measurement precision
If real-time multi-sensor data fusion is performed, then accurate environmental representation is achieved, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary processing of sensor data by transforming LIDAR point cloud data into geo-referenced coordinate space using pre-stored navigation information before fusion. This preliminary geo-referencing and alignment of data from different sensors reduces the computational burden during real-time fusion, as the data is already organized in a common coordinate framework ready for integration.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enhances safety and situational awareness for pilots by providing a clear synthetic image in low or zero visibility conditions, allowing for safe operation of aircraft, and can be applied to various vehicles in obscured atmospheres for obstacle detection and collision avoidance.
Implementation Method 1
generating a first image of the environment using infrared information from an infrared (IR) camera on the vehicle
Implementation Method 2
generating a second image of the environment using laser point cloud data from a LIDAR on the vehicle
Data Source
AI summary
Within examples, systems and methods of generating a synthetic image representative of an environment of a vehicle are described comprising generating a first image using infrared information from an infrared (IR) camera, generating a second image using laser point cloud data from a LIDAR, generating an embedded point cloud representative of the environment based on a combination of the first image and the second image, receiving navigation information traversed by the vehicle, transforming the embedded point cloud into a geo-referenced coordinate space based on the navigation information, and combining the transformed embedded point cloud with imagery of terrain of the environment to generate the synthetic image representative of the environment of the vehicle.