Multimodal Camera Passthrough With Thermal Parallax Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional MR systems fail to optimize passthrough visualizations, particularly in low-visibility environments, leading to poor-quality or useless visualizations.
Innovation Solution
The system employs multiple cameras, including visible light, low light, and thermal imaging cameras, to generate enhanced passthrough visualizations by aligning and combining different types of image data, such as visible light and thermal images, to provide additional information and improve visibility in various lighting conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional single-modality cameras are used for passthrough visualization, then the system complexity is low, but the visualization quality deteriorates in low-visibility environments
Solution Approach 1:
The patent combines multiple camera modalities (visible light camera and thermal imaging camera) into a unified passthrough visualization system. The visible light camera captures standard visual information while the thermal imaging camera captures heat signatures, and both images are merged to create a composite passthrough view that maintains reliability across different visibility conditions.
Solution Approach 2:
The patent creates a composite image by overlaying thermal image data onto visible light passthrough images. This composite visualization combines the strengths of both modalities - the spatial detail of visible light imaging with the thermal contrast information - to produce a robust passthrough experience that works in both normal and low-visibility environments.
2Measurement precision
If visible light cameras are used alone, then the device complexity is low, but the measurement precision deteriorates in low-light conditions
Solution Approach 1:
The thermal imaging camera acts as an intermediary sensor that provides alternative measurement data when visible light is insufficient. Instead of relying solely on the visible light camera which fails in low-light conditions, the system introduces thermal imaging as a complementary measurement modality that can detect heat signatures independent of ambient light levels.
3Illumination intensity
If thermal imaging camera is added to the system, then the visibility in low-light conditions is improved, but the device complexity increases
Solution Approach 1:
The thermal imaging camera serves multiple functions within the MR system: it provides enhanced visibility in low-light conditions, enables detection of heat signatures for improved object identification, and contributes to creating a more robust passthrough visualization that works across diverse environmental conditions.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The enhanced passthrough visualizations provide improved user experience by enabling the user to view and interact with their environment more effectively, even in low-visibility scenarios, by merging and overlaying different types of image data to create a composite image.
Implementation Method 1
a third camera structured to detect long wave infrared radiation
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Enhanced passthrough images are generated and displayed. A current visibility condition of an environment is determined. Based on the current visibility condition, a first camera or a second camera, which detect light spanning different ranges of illuminance, is selected to generate a passthrough image of the environment. The selected camera is then caused to generate the passthrough image. Additionally, a third camera, which is structured to detect long wave infrared radiation, is caused to generate a thermal image of the environment. Parallax correction is performed by aligning coordinates of the thermal image with corresponding coordinates identified within the passthrough image. Subsequently, the parallax-corrected thermal image is overlaid onto the passthrough image to generate a composite passthrough image, which is then displayed.