Mixed Reality Compositing With Real-Time Scene Lighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
3D objects in mixed reality often appear as overlays rather than integral parts of the real-world environment due to unlit or static illumination, breaking the illusion of integration.
Innovation Solution
Estimate real-world scene lighting using device cameras and sensors to infer a 3D lighting configuration for digital objects, replicating real-world conditions in real-time, and apply this lighting to composited 3D objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If 3D objects are composited onto background images in mixed reality, then the objects can be displayed in the virtual environment, but the objects appear as overlays with unlit or static illumination that breaks the illusion of integration
Solution Approach 1:
The patent dynamically changes lighting parameters (intensity, color temperature, direction) of the 3D objects based on the analyzed background image conditions. The system adjusts these parameters in real-time to match the ambient lighting environment, transforming the objects from static, unlit appearances to dynamically lit appearances that integrate seamlessly with the background.
Solution Approach 2:
The system implements a feedback loop where the background image is continuously analyzed to detect lighting conditions, and this information feeds back to adjust the illumination of composited objects. The lighting on objects is continuously adapted based on the detected ambient light characteristics, ensuring consistent integration as the environment or viewing conditions change.
2Reliability
If dynamic lighting analysis is performed on background images to adjust object illumination, then the integration realism is improved, but the computational complexity and processing time increase
Solution Approach 1:
Instead of analyzing the entire background image uniformly, the system focuses lighting analysis on specific regions relevant to the composited objects. It determines ambient light characteristics from local areas of the background image that would logically illuminate the object positions, reducing the computational scope while maintaining accuracy.
Solution Approach 2:
The patent applies partial action by selecting only the essential lighting parameters (ambient intensity, color temperature, primary light direction) from the background analysis rather than computing all possible lighting properties. This selective approach provides sufficient realism for visual integration while significantly reducing computational requirements compared to full physical lighting simulation.
3Reliability
If lighting conditions are inferred from background images to illuminate 3D objects, then the objects appear more integrated with the environment, but the processing time for real-time compositing increases
Solution Approach 1:
The system performs preliminary analysis of the background image to pre-determine ambient lighting conditions before objects are composited. By analyzing the background lighting environment in advance and caching the extracted lighting parameters, the system reduces the processing time required during actual object compositing operations, enabling faster real-time rendering.
Solution Approach 2:
The lighting analysis module serves multiple functions: it not only determines ambient light intensity for object illumination but also extracts color temperature, light direction, and shadow characteristics that can be applied to multiple different objects simultaneously. This multi-functional approach amortizes the processing cost across several compositing operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system may include a memory device to store instructions and data, and at least one processing device to execute the instructions stored in the memory device to: receive a background image and a digital object to be composited onto the background image in a mixed reality view, generate a 2D bounding region for the digital object, select a version of the background image at a pre-defined resolution, overlay the 2D bounding region on the selected version of the background image and obtain a set of samples of the colors of pixels of the selected version along a perimeter of the 2D bounding region, and determine a value for one or more digital lighting sources to illuminate the digital object in the mixed reality view, based, at least in part, on the set of samples.