Dynamic Image Registration for Multi-Modal Sensor Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current night-vision sensors for military use operate separately, leading to redundant equipment, decreased efficiency, and increased vulnerability due to the inability to seamlessly transition between disparate sensor modalities, and existing Inertial Measurement Units (IMUs) are prone to interference from environmental magnetic fields, causing inaccuracy.
Innovation Solution
A system that synchronizes imagery between video camera sensors using Inertial Measurement Units (IMUs) and image feature registration through processing, allowing for real-time calibration and correction of camera orientations, and synergistically combining IMU and image feature information to maintain accurate registration despite environmental noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple sensor modalities (thermal and electro-optical) are used separately, then each sensor can function independently, but the system complexity increases and operational efficiency decreases due to redundant equipment and inability to seamlessly transition between sensors
Solution Approach 1:
The patent combines multiple sensor modalities (thermal and electro-optical cameras) into a single integrated system with a unified display interface. The images from both sensors are registered and fused together, allowing the user to view both modalities simultaneously or transition seamlessly between them through a single device rather than switching between separate equipment.
Solution Approach 2:
The integrated sensor system provides multi-functional capability by incorporating both thermal and electro-optical sensing modalities within a single platform. The system can operate in various modes including viewing only electro-optical imagery, only thermal imagery, or a combination of both, making it adaptable to different operational requirements while eliminating the need for separate dedicated devices.
2Measurement precision
If a rifle-mounted thermal weapon sight is used to accurately aim, then the soldier can precisely direct their rifle, but the soldier loses situational awareness of the broader environment due to the narrow field-of-view
Solution Approach 1:
The patent overlays the narrow thermal weapon sight imagery onto the broader electro-optical camera field-of-view, creating a multi-layered visual display. This allows the precise thermal aim point information to be superimposed on the wider environmental context, enabling the user to maintain both situational awareness and aiming precision simultaneously through a single unified view.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables seamless and accurate integration of multiple sensor modalities, reducing user distraction and enhancing situational awareness by providing continuous, unified visual input without the need for equipment rearrangement, while minimizing the impact of IMU drift and noise.
Implementation Method 1
The accelerometers are used to determine the vertical direction, by measuring the acceleration of gravity.
Implementation Method 2
The magnetometers use the Earth's magnetic field to determine magnetic North, and thus left-to-right orientation.
Implementation Method 3
Finally, the gyroscopes are used to stabilize the instantaneous measurements from the other devices, and neutralize jitter from sensor noise.
Data Source
AI summary
A methodology and apparatus is described that registers images outputted by at least two video camera sensors that are not necessarily bore-sighted nor co-located together. Preferably this can be performed in real-time at least at video rate. The two video camera sensors can be either of similar or two different modalities (e.g., one can be intensified visible, while the other can be thermal infrared) each possibly with a different field-of-view. The methodology and apparatus take advantage of a combination of Inertial Measurement Units (IMUs) information and image registration between the two camera sensors from computational image processing. In one preferred embodiment, the system uses a rifle mounted weapon sight camera sensor and a helmet mounted sensor camera. The aim point of the weapon sight camera sensor (e.g., typically at the image center) zeroed to the true rifle aim point is digitally overlayed as a reticle on the video image from the helmet sensor, displayed on a head mounted display (HMD). In another embodiment pixel-level image fusion is displayed across the overlapping field-of-view of the rifle mounted camera sensor weapon sight with the helmet mounted camera image.