Light Estimation Model for AR Virtual Object Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current light estimation techniques in augmented reality and computer graphics struggle to accurately represent light information, leading to unnatural rendering of virtual objects in varying illuminance conditions, as they rely on variable ISP settings that can mismatch actual light conditions.
Innovation Solution
A method involving a light estimation model trained using a combination of reference and background images captured with fixed and variable ISP settings, where the model estimates integrated light information and renders virtual objects based on this data, incorporating simultaneous localization and mapping information to improve accuracy across different positions and directions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If variable ISP settings are used to adapt to different lighting conditions, then the system can handle varying illuminance conditions, but the light estimation accuracy deteriorates because the ISP settings may mismatch actual light conditions
Solution Approach 1:
The patent applies preliminary action by capturing reference images with fixed ISP settings before actual use. These reference images serve as pre-computed benchmarks that allow the system to estimate accurate light information without relying on potentially inaccurate variable ISP settings during actual operation. The fixed ISP settings in the reference images provide a stable foundation for training the light estimation model.
Solution Approach 2:
The patent uses copying by creating a reference image that replicates the actual scene under known lighting conditions. This reference image serves as a copy of the ground truth light information, allowing the system to compare and estimate accurate light parameters without being influenced by the variable ISP settings applied during actual capture.
2Measurement precision
If fixed ISP settings are used to maintain consistent processing, then light estimation accuracy can be maintained, but the system's adaptability to different lighting conditions deteriorates
Solution Approach 1:
The system performs preliminary capture of reference images using fixed ISP settings across multiple lighting conditions. This preliminary action creates a comprehensive dataset that pre-adapts the system to various illuminance conditions, allowing accurate light estimation to be maintained while achieving adaptability through the pre-computed reference data.
Solution Approach 2:
The patent applies parameter changes by varying the ISP settings during reference image capture to cover different lighting conditions, while maintaining fixed settings during the actual light estimation process. This allows the system to adapt to different lighting conditions through the reference data while maintaining measurement precision through consistent processing during operation.
3Adaptability or versatility
If multiple reference images with different ISP settings are used for training, then the light estimation model can handle varying conditions, but the training complexity and data processing requirements increase
Solution Approach 1:
The patent applies segmentation by dividing the training process into separate phases: capturing reference images with fixed ISP settings, processing these references to create training data, and then using this data to train the light estimation model. This segmentation allows the system to handle varying lighting conditions through structured processing while managing training complexity through systematic data preparation.
Solution Approach 2:
The patent uses copying by creating synthetic training data from reference images. Instead of directly processing complex variable ISP settings during training, the system creates simplified copies of the reference images that can be systematically processed to generate training datasets, reducing the immediate complexity of training while still capturing varying lighting conditions.
Data Source
AI summary
A method and device with light estimation are provided. A method performed by an electronic device includes generating a reference image based on image data acquired by capturing a reference object and based on a first image signal processing (ISP) setting, generating a background image based on raw image data acquired by capturing a real background in which the reference object is positioned and based on a second ISP setting, estimating light information corresponding to the background image using a light estimation model, rendering a virtual object image corresponding to the light information and the reference object, and training the light estimation model based on a difference between the reference image and the virtual object image.