Reference Object Shadow Maps for Real-Time 3D Lighting Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Estimating accurate lighting conditions in mixed reality applications is a time-consuming task, especially for real-time renderings, and techniques are needed to simplify this process to provide realistic renderings on devices including mobile devices.
Innovation Solution
Using a reference object with known 3D geometry, such as a standard microphone or smart connected object, to estimate lighting by analyzing shadows cast on a planar surface, eliminating the need for additional static cameras and reducing latency through precomputed shadow maps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional lighting estimation techniques are used in mixed reality applications, then accurate lighting conditions can be estimated, but the process is time-consuming and not suitable for real-time rendering
Solution Approach 1:
The system pre-computes shadow maps for a reference object from multiple virtual viewpoints and stores them in advance. When lighting estimation is needed, the system simply retrieves and compares these pre-computed shadow maps with captured images, eliminating the need for time-consuming real-time shadow rendering and enabling rapid lighting parameter estimation.
Solution Approach 2:
The system creates virtual copies of shadow maps from a reference object at predetermined viewpoints and stores them for later comparison. By copying and comparing these pre-generated shadow patterns with actual captured images, the system可以快速 estimate lighting parameters without performing complex real-time shadow calculations.
2Reliability
If virtual lighting models are used to create hard shadows for realistic rendering, then shadow realism is improved, but the rendering process becomes time-consuming and difficult
Solution Approach 1:
Shadow maps representing hard shadows from the reference object are pre-computed and stored in advance for multiple virtual viewpoints. This eliminates the need to generate complex shadow patterns in real-time, enabling both realistic shadow rendering and fast processing by simply retrieving and comparing pre-computed shadow data.
Solution Approach 2:
The system extracts only the essential shadow information from pre-computed shadow maps of a reference object, separating the shadow pattern extraction from the full rendering process. This allows the system to obtain realistic shadow characteristics without performing complete time-consuming rendering calculations.
3Measurement precision
If additional static cameras are used to capture lighting information, then lighting estimation accuracy is improved, but device complexity and hardware requirements increase
Solution Approach 1:
The system uses the device's existing camera to capture images of the reference object and performs lighting estimation by comparing captured images with pre-computed shadow maps. The device serves its own lighting estimation needs using only its built-in camera, eliminating the requirement for additional static cameras or specialized hardware.
Solution Approach 2:
The existing camera in the device is used for multiple purposes: capturing general scene images and also capturing reference images for lighting estimation. This multi-functional use of the existing camera eliminates the need for dedicated lighting measurement hardware, reducing device complexity while maintaining estimation accuracy.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables rapid and accurate lighting estimation without additional hardware, allowing for seamless integration into augmented, mixed, or virtual reality experiences on mobile devices.
Implementation Method 1
analyzing shadows cast on a planar surface
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
A method and device is provided for processing images. In one embodiment, the method comprises receiving an image is received of a scene having a plurality of real objects and determining if at least one of these real objects is a reference object stored in a database. Subsequently, candidate shadow map of the reference object identified is retrieved when available and when a reference object cannot be identifiled, an object in the scene with adequate parameters is selected and the candidate shadow maps of this new reference object using said lighting parameters is computed.