3D Object Light Estimation for Realistic AR Shading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality (AR) systems struggle to render virtual objects realistically due to inconsistencies between the texture and brightness of virtual content and the lighting conditions of the physical scene, leading to a sense of unnaturalness.
Innovation Solution
A machine learning system estimates light conditions using HDRI environment maps and 3D scans to predict lighting parameters, which are applied to virtual objects through a physically based rendering shader, ensuring consistency with the physical environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Illumination intensity
If traditional rendering methods are used for virtual objects, then the rendering process is simple and fast, but the lighting consistency with the physical scene deteriorates, making the virtual content appear unrealistic
Solution Approach 1:
The patent introduces an intermediary lighting estimation system that analyzes the physical scene and generates lighting parameters, which then mediate between the real environment and virtual object rendering. This intermediary layer ensures lighting consistency without requiring complex ray-tracing or global illumination calculations, resolving the contradiction between realism and system complexity.
Solution Approach 2:
The system changes the rendering parameters by estimating lighting conditions (illumination direction, intensity, color temperature) from the physical scene and applying these parameters to the virtual objects. This parameter-based approach achieves realistic lighting effects without the computational complexity of physics-based rendering methods.
2Illumination intensity
If complex lighting estimation methods are used, then lighting consistency improves, but the computational time and processing power increase
Solution Approach 1:
The patent replaces complex mechanical/computational lighting calculation systems with a machine learning-based estimation system. The pre-trained neural network rapidly estimates lighting parameters from scene images, achieving accurate lighting results in real-time without the computational burden of traditional physics-based rendering methods.
Solution Approach 2:
The lighting estimation model is pre-trained offline with large datasets of rendered images and their corresponding lighting parameters. This preliminary action transfers complex computational work to the training phase, enabling fast real-time inference during actual AR rendering without sacrificing lighting accuracy.
3Productivity
If simple rendering is used, then processing speed is fast, but the realism and immersion of the AR experience deteriorates
Solution Approach 1:
The system employs self-service mechanisms where the mobile device's existing camera and processing capabilities are utilized for lighting estimation. The device uses its own imaging sensors to capture the physical scene and automatically estimates lighting parameters without requiring external specialized equipment, maintaining fast processing speeds while improving realism.
Data Source
AI summary
A method is disclosed comprising accessing an image; identifying a virtual object corresponding to a physical object depicted in the image; and determining shading parameters for the virtual object based on a machine learning model. The model is trained by generating a synthetic face image using a first renderer; predicting lighting parameters from the synthetic face image with a neural network; generating a predicted sphere image using a second renderer based on the predicted lighting parameters; generating a synthetic sphere image using a third renderer; comparing the predicted sphere image with the synthetic sphere image; and training the neural network based on the comparison. The method further comprises generating a shaded virtual object by applying the shading parameters to the virtual object and displaying the shaded virtual object as a layer over the image.