AR Face Illumination Estimation for Virtual Object Lighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality (AR) technologies face challenges in achieving consistent lighting for virtual objects integrated into real environments, especially in environments with unknown and arbitrary lighting conditions.
Innovation Solution
The method involves capturing an image of a user's face with a user-facing camera, estimating the head pose, and then using a facial illumination estimator to reconstruct the face-incident light. This light is used to render virtual objects with consistent illumination, blending them seamlessly into the real environment using differential rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Illumination intensity
If additional setups like mirrors or special cameras are used to estimate environment light, then lighting consistency for virtual objects is improved, but device complexity and applicability are worsened
Solution Approach 1:
The system uses the existing front-facing camera to capture images of the user's face, which already contains illumination information. By analyzing the face image directly, the system extracts light direction and intensity without requiring any additional light-capturing devices or mirrors. The face itself serves as the light probe, eliminating the need for separate measurement equipment.
Solution Approach 2:
The front-facing camera serves dual purposes: capturing the user's face for AR overlay positioning and simultaneously capturing illumination information for lighting estimation. This multi-functional use of the existing camera eliminates the need for dedicated light measurement devices while maintaining lighting consistency.
2Illumination intensity
If panoramic images or mirror sphere images are used to capture environment light, then lighting estimation is improved, but device complexity and scene alteration are worsened
Solution Approach 1:
The user's face acts as a self-contained light probe that directly reveals the illumination characteristics of the environment. By analyzing how light interacts with the face's known geometry and material properties, the system extracts accurate lighting information without needing to capture panoramic views or use mirror spheres.
Solution Approach 2:
The system extracts only the essential illumination information (light direction, intensity, and color) from the face image, separating this critical data from the rest of the scene. This extraction approach provides sufficient lighting estimation without capturing or processing the entire environmental panorama.
3Measurement precision
If common approaches are used to estimate environment light directions, then lighting direction is improved, but light source position information is lost
Solution Approach 1:
The system uses the known 3D geometry and material properties of the face as a reference to interpret the captured face image. By comparing the observed lighting patterns on the face with predictions from light transport models, the system iteratively refines estimates of both light direction and light source position, with each informing the other to achieve accurate results.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The invention relates to a method of representing a virtual object in a view of a real environment which comprises the steps of providing image information of a first image of at least part of a human face captured by a first camera, providing at least one human face specific characteristic, determining at least part of an image area of the face in the first image as a face region of the first image, determining at least one first light falling on the face according to the face region of the first image and the at least one human face specific characteristic, and blending in the virtual object on a display device in the view of the real environment according to the at least one first light. It further comprises providing a plurality of face sample positions, providing, for each of the plurality of face sample positions, a radiance transfer function, determining image positions of at least part of the face sample positions in the first image, and determining the at least one first light according to intensity information associated with the image positions of at least part of the face sample positions in the first image and the radiance transfer functions associated with the at least part of the face sample positions in the first image. The invention also relates to a system for representing a virtual object in a view of a real environment.