XR Virtual Content Tone Map Adjustment for Pass-Through Lighting Consistency

The virtual content tone map is dynamically adjusted based on the pass-through video tone map. The system continuously adapts the virtual content's luminance values to match the lighting conditions captured by the image sensor, allowing the tone map to change over time as the user moves through different environments. This resolves the contradiction by making the previously static tone map adaptive to changing real-world lighting conditions.

The system changes the parameters of the virtual content tone map by aligning its luminance values with those of the pass-through video tone map. Specifically, the brightest pixel exposure values are used as reference points for rescaling, and the tone map curves are adjusted to match between virtual and real content. This parameter alignment ensures that virtual objects appear lit by the same environmental light sources as the captured video, resolving the lighting inconsistency issue.

The system uses the pass-through video tone map as an intermediary to transfer lighting information from the real world to virtual content. Instead of directly modeling complex environmental lighting, the patent leverages the existing ISP tone map processing pipeline as a mediator that captures real-world lighting characteristics and applies them to virtual content. This intermediary approach simplifies the overall system by reusing existing image processing infrastructure.

The system copies the luminance characteristics from the pass-through video tone map and applies them to the virtual content tone map. By replicating the exposure values, midtone positions, and highlight crush characteristics from the real-world capture pipeline, the patent achieves lighting consistency without requiring complex new processing algorithms. This copying strategy leverages the already-optimized ISP pipeline to drive virtual content rendering.