Image Rendering With Light Source Color Prediction for Mixed Reality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mixed reality systems face challenges in accurately merging virtual content with real scenes due to color deviations caused by inadequate prediction of light source colors, particularly in optical see-through MR devices, and lack of fine control over lighting estimation in various indoor and outdoor environments.
Innovation Solution
An image processing model that predicts light source color information and panoramic images using a block-based approach, decoupling lighting and scene features, and employs attention mechanisms to enhance accuracy, enabling high-resolution lighting map generation and color correction for consistent virtual reality fusion across different scenes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If light source color prediction is not performed accurately, then the processing speed is maintained, but color deviation occurs between rendered image and real scene image
Solution Approach 1:
The patent divides the image processing into multiple networks: an encoding network that extracts features, a first prediction network that predicts light source color information, and a decoding network that reconstructs the panoramic image. This segmentation allows each network to specialize in specific tasks, improving prediction accuracy while maintaining manageable complexity through modular architecture.
Solution Approach 2:
The patent introduces encoded feature maps as an intermediary between the input image and the prediction output. The encoding network transforms the input image into compressed feature representations, which are then processed by the prediction network. This intermediary representation captures essential lighting information efficiently, improving prediction accuracy without requiring the full complexity of processing the entire high-resolution image.
2Manufacturing precision
If high-resolution panoramic image and light source color information are predicted simultaneously, then the virtual reality merging quality is improved, but the processing time increases
Solution Approach 1:
The patent performs preliminary feature extraction through the encoding network before prediction. By pre-processing the input image to extract relevant features and compress them into encoded feature maps, the system prepares the data in advance for prediction. This preliminary action reduces the computational burden during the actual prediction phase, enabling simultaneous high-resolution panoramic image and light source color prediction while controlling processing time.
Solution Approach 2:
The patent transforms the input image into a different parameter space through encoding, representing the image as compressed feature maps rather than raw pixel data. This parameter transformation allows the prediction network to work with more compact representations, extracting both panoramic and lighting information more efficiently. The encoded features capture essential information in a form that accelerates subsequent prediction operations.
3Measurement precision
If block-based processing with attention mechanisms is used, then the lighting estimation accuracy is improved, but the computational complexity increases
Solution Approach 1:
The patent divides the image into multiple blocks and processes each block independently through the encoding and prediction networks. This block-based segmentation allows parallel processing of different image regions, improving lighting estimation accuracy by capturing local variations while enabling efficient computation through distributed processing across multiple blocks simultaneously.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A processor-implemented method includes obtaining an input image, predicting light source color information of a scene corresponding to the input image and a panoramic image corresponding to the input image using an image processing model, and generating a rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image.