Local Illumination Map Training for Position-Aware AR Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models struggle to accurately predict spatially varying illumination conditions for different positions within a scene, leading to unrealistic rendering of virtual objects in augmented reality applications, as they are typically trained with a single set of illumination conditions for the entire scene.
Innovation Solution
A method for generating a large training dataset by obtaining training images with a first camera and determining local illumination maps for multiple positions, transforming these maps to a common coordinate system, and associating them with reference data to create a training dataset that accounts for varying illumination conditions across the scene.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single reference object is used to estimate illumination conditions, then the device complexity is reduced, but the illumination conditions cannot be accurately predicted for different positions relative to the scene
Solution Approach 1:
The patent divides the scene into multiple regions of interest, each with its own reference object. Instead of using a single reference object for the entire scene, the method segments the illumination estimation task into multiple local estimations, where each reference object provides illumination data for its specific region. This resolves the contradiction by maintaining low device complexity (using minimal reference objects) while improving measurement precision through localized illumination prediction.
Solution Approach 2:
The patent applies local quality by allowing different parts of the scene to have different illumination characteristics. Each reference object captures local illumination conditions specific to its position, enabling the system to predict spatially varying illumination conditions accurately without requiring a complex array of reference objects throughout the scene.
2Measurement precision
If multiple gray spheres are manually positioned in the scene to capture spatially varying light conditions, then the illumination conditions for different positions can be estimated, but the productivity decreases due to the large amount of manual work required
Solution Approach 1:
The patent uses a single reference object that can be digitally copied and transformed to multiple positions through coordinate system transformations. Instead of physically placing multiple gray spheres in the scene, the system captures illumination data from one reference object and mathematically transforms this data to represent illumination conditions at multiple positions. This resolves the contradiction by maintaining measurement precision through multiple position estimations while dramatically improving productivity by eliminating manual repositioning work.
Solution Approach 2:
The patent performs preliminary action by capturing all necessary illumination data in a single scene configuration with one reference object. The coordinate system transformations and illumination map generation are performed computationally after the single capture, rather than requiring multiple physical setups. This approach maintains accurate spatially varying illumination estimation while significantly improving dataset generation efficiency.
3Device complexity
If a single set of illumination conditions is applied to the whole scene, then the device complexity is reduced, but the rendering realism deteriorates under spatially varying light conditions
Solution Approach 1:
The patent segments the illumination model into multiple local illumination maps, each corresponding to a specific region or position in the scene. Instead of applying a single global illumination set, the system generates and applies localized illumination conditions to different parts of the scene, improving rendering realism under spatially varying light conditions while keeping the overall device complexity manageable through systematic processing.
Solution Approach 2:
The patent introduces dynamics by making the illumination conditions adaptive to position within the scene. The system dynamically selects and applies appropriate illumination conditions based on the virtual object's position relative to the scene, using coordinate system transformations to determine the correct local illumination map. This resolves the contradiction by improving rendering realism through position-dependent illumination while maintaining reasonable device complexity through automated selection processes.
Data Source
AI summary
An image processing method generates a training dataset for training a machine learning model to predict illumination conditions for different positions relative to a scene, the training dataset including training images and reference data. The method includes: obtaining a training image of a training scene acquired by a first camera having an associated first coordinate system; determining local illumination maps associated to a respective position in the training scene in a respective second coordinate system and representing illumination received from different directions around the position; transforming the position of each local illumination map from the second to the first coordinate system; responsive to determining that the transformed position of a local illumination map is visible: transforming the local illumination map from the second to the first coordinate system and including the transformed local illumination map and its transformed position in the reference data associated to the training image.