Photorealistic AR Rendering via Lighting and Occlusion Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for creating rendered images of synthetic objects for augmented reality fail to accurately represent lighting and material properties in real-time, leading to inaccurate integration with the environment, particularly in terms of shading, occlusion, and dynamic shadows, which limits their effectiveness in applications like design and purchasing decisions.
Innovation Solution
A computing device generates a photorealistic augmented reality experience by obtaining a 3D representation of the environment, determining lighting, rendering 3D assets with accurate shading and lighting, and creating an occlusion mask to composite the assets with the environment, ensuring accurate occlusion and lighting matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional rendering methods are used to create images of synthetic objects for augmented reality, then processing speed is maintained, but lighting accuracy and material property representation deteriorate
Solution Approach 1:
The system performs preliminary actions by capturing environment images and analyzing lighting conditions before rendering the synthetic object. The lighting representation is determined in advance from the captured images, allowing the rendering process to apply accurate lighting and shading without real-time computation delays. This preliminary analysis of the environment's lighting setup enables photorealistic rendering while maintaining efficient processing speeds.
2Manufacturing precision
If traditional rendering methods are used, then processing time is reduced, but occlusion representation with the background deteriorates
Solution Approach 1:
The system generates an occlusion mask in advance by comparing the rendered synthetic object with the captured environment image. This preliminary occlusion analysis identifies which parts of the background should be visible and which should be occluded by the synthetic object. By performing this occlusion determination before final composite image generation, the system achieves accurate occlusion representation without adding perceptible processing delay to the augmented reality experience.
3Measurement precision
If traditional methods are used to render synthetic objects, then device complexity is maintained, but photorealistic quality deteriorates
Solution Approach 1:
The system introduces several intermediary elements to achieve photorealistic quality without excessive complexity: (1) an environment image capture module that mediates between the physical environment and digital rendering, (2) a lighting representation module that translates captured images into lighting parameters, and (3) an occlusion mask generator that mediates between the synthetic object and background. These intermediaries bridge the gap between simple rendering and photorealistic quality while keeping the overall system manageable.
Data Source
AI summary
A digital medium environment includes at least one computing device (e.g., a user's mobile device) in an environment. Systems and techniques are described for presenting a photorealistic augmented reality experience in real time on the computing device. A lighting representation of the environment is obtained from a partial view of the environment, such as from an image of the environment, and used to render a 3D asset (e.g., graphics of a synthetic object). The 3D asset is inserted into the environment by compositing the rendered 3D asset with the image of the environment accounting for occlusion and based on the lighting representation to form a composite image. Photorealistic processing is applied to the composite image by adding lighting for the rendered 3D asset to a portion of the background of the composite image. Consequently, the inserted graphic is indistinguishable from the real-world scene.