HMD Lighting Map Generation With Ambient Sensor and Tracking Camera
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing head-mounted display (HMD) apparatuses fail to accurately capture and render virtual-reality (VR) objects in mixed-reality environments due to improper illumination, missing reflections, and the need for users to repeatedly scan their surroundings when lighting conditions change, which is inefficient and user-unfriendly.
Innovation Solution
Integrate an ambient light sensor, a tracking camera, and a video-see-through camera with a processor to capture and reconstruct a lighting map with correct total light, tone, and distribution, using the ambient light sensor for direction and tracking camera for illumination data, and the VST camera for color data, without requiring special lenses or machine-learning algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Area of stationary object
If HDR visible-light cameras are used to capture light from multiple directions, then the field of view coverage is improved, but the device complexity and cost increase significantly
Solution Approach 1:
The patent divides the lighting capture function into two separate components: a monochromatic tracking camera that captures intensity information from a wide field of view, and an ambient light sensor that measures overall illumination levels. This segmentation allows each component to be optimized for its specific function rather than requiring a complex HDR camera system to handle all lighting aspects.
Solution Approach 2:
The patent repurposes the monochromatic tracking camera, originally designed for eye-tracking functionality, to also capture lighting information. This multi-functional use of existing hardware eliminates the need for additional specialized cameras while still achieving comprehensive lighting map generation across the entire field of view.
2Area of stationary object
If monochromatic tracking cameras are used to capture total incident light, then the field of view coverage is improved, but the color accuracy deteriorates due to saturation of direct lights
Solution Approach 1:
The patent merges the data from two different sensors: the monochromatic tracking camera provides spatial distribution and intensity information across the field of view, while the ambient light sensor provides accurate overall illumination measurements. By combining these complementary data sources, the system achieves both wide field of view coverage and accurate lighting representation including color information.
Solution Approach 2:
The patent introduces computational processing as an intermediary that fuses the monochromatic intensity data with ambient light sensor measurements. This computational mediation allows the system to reconstruct accurate color and intensity information by combining the strengths of both sensors while compensating for their individual limitations.
3Loss of information
If users scan the real-world environment by rotating their head, then the lighting information coverage is improved, but the ease of operation deteriorates
Solution Approach 1:
The patent creates a dynamic lighting map that automatically updates in real-time as the user moves their head or as lighting conditions change in the environment. The system continuously captures new lighting data and updates the virtual lighting without requiring the user to perform deliberate scanning actions, making the process seamless and automatic rather than manual and deliberate.
Solution Approach 2:
The patent implements feedback mechanisms where the system monitors changes in the real-world lighting environment and automatically adjusts the virtual lighting map accordingly. Sensors detect lighting changes and trigger automatic updates to the lighting map, providing continuous feedback loops that maintain accuracy without user intervention.
4Measurement precision
If time-warping is applied to HDR bracketed frames, then the illumination accuracy is improved, but the visual quality deteriorates due to choppy frames and uneven movement
Solution Approach 1:
The patent performs preliminary integration of monochromatic intensity data with ambient light sensor measurements to create a complete lighting map before rendering virtual objects. By having the lighting information ready in advance and properly integrated, the system eliminates the need for post-capture time-warping operations that would compromise visual quality.
Solution Approach 2:
The patent replaces the mechanical/time-based HDR bracketing approach with a computational fusion approach that combines sensor data in real-time. Instead of capturing multiple frames at different exposures and then time-warping them, the system uses complementary sensors to directly measure and integrate lighting information, eliminating the need for time-warping operations entirely.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables a realistic and immersive mixed-reality experience by continuously improving the lighting map, eliminating the need for users to re-scan their environment, and ensuring accurate illumination and reflection rendering.
Implementation Method 1
capture, using the ambient light sensor, an average red-green-blue (RGB) illuminance (A) of a given region of a real-world environment
Implementation Method 2
capture, using the tracking camera, a grayscale image of the real-world environment, wherein values of pixels in the grayscale image indicate intensities (i) of the pixels
Data Source
AI summary
Disclosed is head-mounted display (HMD) apparatus with an ambient light sensor (ALS), tracking camera (TC), video-see-through (VST) camera, and processor(s) configured to: capture average red-green-blue (RGB) illuminance (A) of region of real-world environment (RE) from HMD pose; capture grayscale image (GI) of RE from HMD pose, wherein pixel values indicate intensities (i); calculate average TC intensity (I) for part of FOV of TC that overlaps with FOV of ALS, based on intensities (i); calculate average illuminance (J) for said part of FOV of TC that overlaps with FOV of ALS, based on intensities (i) and pre-determined response (r) of ALS; calculate corrected average RGB illuminance (AC) for region from HMD pose, based on A and ratio of I and J; and determine RGB illuminance for each pixel of GI, based on AC, ratio of intensity (i) of said pixel in GI to I.