AR Lighting Estimation Using Multi-Camera Environment Texture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing light estimation techniques for augmented reality (AR) are inefficient, either requiring extensive offline processing or having high runtime costs, and struggle to accurately match rendered objects with real-world lighting conditions.
Innovation Solution
A method for rendering AR objects using an electronic device that captures images from both front- and rear-facing cameras, combines them to create an environment texture, determines light source positions, and renders the object based on this texture and light source data, allowing for more realistic lighting conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If extrapolation techniques are used to estimate lighting conditions, then the accuracy of lighting estimation is improved, but the runtime cost increases and extensive offline processing is required
Solution Approach 1:
The patent performs preliminary actions by capturing images from multiple cameras (front-facing and rear-facing) and processing them to create an environment texture and identify light sources before the actual AR rendering occurs. This offline preparation reduces the runtime computational burden during real-time AR operations.
Solution Approach 2:
The patent segments the lighting estimation process into distinct components: capturing images from multiple cameras, combining them to create an environment texture, identifying light sources, and using this pre-computed data for rendering. This segmentation allows each component to be optimized independently and reduces overall runtime cost.
2Loss of time
If approximation techniques are used to estimate lighting conditions, then the runtime cost is reduced, but the accuracy of lighting estimation deteriorates
Solution Approach 1:
The patent merges the advantages of both approximation and extrapolation techniques by combining multiple camera inputs (front-facing and rear-facing cameras) to create a comprehensive environment texture. This merging approach provides accurate lighting estimation while maintaining reasonable runtime performance through efficient image processing.
Solution Approach 2:
The patent introduces an intermediary approach by using an environment texture as a mediator between the captured images and the final rendering. This environment texture serves as a pre-computed lighting reference that bridges the gap between simple approximation and complex extrapolation, providing balanced accuracy and performance.
3Productivity
If existing light estimation techniques are used, then the basic rendering function is achieved, but the realism of the composite view deteriorates
Solution Approach 1:
The patent applies local quality by identifying specific light sources in the environment and using this information to locally adjust the rendering of AR objects. Instead of using uniform lighting approximation, the system adapts lighting conditions specifically at the locations where light sources are detected, significantly improving realism.
Solution Approach 2:
The patent implements feedback by using the identified light source positions and environment texture information to continuously refine the rendering process. The system uses the captured environmental data as feedback to adjust and optimize the AR object rendering, ensuring realistic lighting matching.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A method for operating an electronic device for rendering an augmented reality object is provided. The method includes obtaining at least one first image of an environment in which an augmented reality virtual object is to be displayed, obtaining an environment texture based on the obtained at least one first image, identifying at least one light source in at least one second image by identifying at least one cluster comprising pixels with a brightness value greater than a threshold, determining a position of the at least one light source based on a position of the pixels within the at least one cluster, rendering the augmented reality virtual object based on the obtained environment texture and the determined position of the at least one light source, and displaying the rendered augmented reality virtual object on a display of the electronic device.