Colorized Passthrough Image Merging Camera Modalities
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mixed-reality systems, including VR and AR, face limitations in generating enhanced passthrough images that effectively combine data from multiple camera modalities, leading to potential disorientation and loss of information when transitioning between environments.
Innovation Solution
The system generates an enhanced image by identifying common pixels between images from different camera modalities, computing alpha maps based on saliency values, and merging textures to create a single colorized image that preserves spatial information and indicates the source of each pixel's texture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If multiple camera modalities are used to capture environmental data, then the completeness and quality of environmental information is improved, but the complexity of processing and integrating data from different modalities increases
Solution Approach 1:
The patent combines data from multiple camera modalities (visible light, thermal, low light) into a unified enhanced image through pixel-level merging. Common pixels are identified across modalities, and textures are merged based on edge detection weights and saliency values, creating a single integrated image that preserves information from all modalities while reducing processing complexity.
Solution Approach 2:
The patent segments the image processing into distinct stages: generating individual modality images, identifying common pixels, computing edge detection weights, determining texture contributions based on saliency, and merging textures. This segmentation allows complex multi-modality processing to be broken down into manageable steps.
2Reliability
If passthrough images are used to show real-world environment, then user orientation and safety are improved, but the image quality and detail accuracy deteriorate
Solution Approach 1:
The patent merges data from multiple camera modalities to produce an enhanced passthrough image that maintains both reliability and detail accuracy. By combining visible light, thermal, and low light camera data at the pixel level with proper weighting based on edge detection and saliency, the system achieves improved image quality while preserving the real-world environmental context needed for user orientation.
3Loss of information
If textures are merged from multiple images to create enhanced image, then the information completeness is improved, but the visual clarity and distinguishability of pixel sources deteriorates
Solution Approach 1:
The patent applies color coding to distinguish the sources of different pixel textures in the enhanced image. Different camera modalities are represented by different colors or color patterns, allowing users to visually identify which modality contributed which pixel information while maintaining the merged image's overall clarity and information completeness.
Data Source
AI summary
Techniques for generating an enhanced image. A first image is generated using a camera of a first modality, and a second image is generated using a camera of a second modality. Pixels that are common between the two images are identified. An alpha map is generated. The alpha map reflects edge detection weights that are computed for the common pixels based on saliency values. A determination is made as to how much texture from the images to use to generate an enhanced image. This determination is based on the edge detection weights included within the alpha map. Based on the edge detection weights, textures are merged from the common pixels to generate the enhanced image. Color is also added to the enhanced image, where the color reflects an additional property (e.g., the texture source for the pixel) that is associated with one or both of the images.